This patch contains the following:
- Initial async VFS code (for reading / writing files)
- Simplified Http Client code
- Added async support to Http Client, RequestHandler
- Improved RequestHandler logging
- Deferred bundle loading to EasyAssetPatch
- Make GetManifestJson run async
This comes with a number of benefits:
- When downloading bundles, it will mention which files succeeded or failed to download
- Bundle loading happens in the initial screen, not the white screen
- Fixed the issue where bundle loading could break bepinex loading (too long load time)
- Modders can now make async http request and read/write files async
I removed logging of sessionid inside the RequestHandler for each request, sessionid is already visible from bepinex log startup parameters.
At last, sorry for the amount of commits it took. I initially wanted to target the 3.9.0 branch, but decided to use 3.8.1 instead as async request can really help out some mods.
Reviewed-on: SPT-AKI/Modules#117
Co-authored-by: Merijn Hendriks <merijn.d.hendriks@gmail.com>
Co-committed-by: Merijn Hendriks <merijn.d.hendriks@gmail.com>
This PR won't affect modders who have already updated their mods to 3.8.0. They just need to re-compile.
This is a resubmission of my previous PR (SPT-AKI/Modules#99) with additional code cleanup and fixes.
Instead of outright removing the functionality, this time I deprecate it instead (marked for removal in next release).
Requires SPT-AKI/Server#274 to function.
## Overview
- HTTP modernization
- Adds `Aki.Common.Http.Client`, a replacement for `Aki.Common.Http.Request` and builds on top of `System.Net.Http.HttpClient`
- Implements failsafe retries when requesting during busy connections
- Improved debugging
- Improved performance
- Deprecades old request code
- Bundle system
- Fixes remote downloaded bundles using external IP
- Implements functional bundle caching from remote sources
- Implements multi-threaded bundle downloads
- Implements Unity-compatible bundle format support
- Extensive cleanup
- Deprecated unneccecary models
## Why?
In it's current state, the bundle system is ducktaped together in 2021, fundumentally broken and in desperate need of a cleanup and fixes.
The HTTP code hasn't been updated since 2021, and `HttpWebRequest` has been deprecated by Microsoft for a while now.
There was also a lot of opportunity left for simple performance gains that even reduces the complexity of the code.
As for why not two separate PRs (HTTP modernization, bundle rework): both were deeply interconnected. A change in one requires modification in the other. Hence the current approach.
## Testing
The code has been validated and tested by @TheSparta and me.
A large section of the code has been implemented and tested extensively by modders.
### Local
1. Start the game from 127.0.0.1
2. The game starts loading bundles from the mods path
### Remote, full re-aquire
1. Start the server from LAN IP (http.json, set host to `cmd > ipconfig` address, example: `192.168.178.32`)
2. Start the game from LAN IP
3. A folder named `user/cache/bundles` is created
### Remote, partial-aquire (deleted)
1. Ensure all bundles are cached
2. Delete one of the aquired bundles from cache
3. Start the server from LAN IP (http.json, set host to `cmd > ipconfig` address, example: `192.168.178.32`)
4. Start the game from LAN IP
5. The bundle is redownloaded
### Remote, partial-aquire (invalid crc)
1. Ensure all bundles are cached
2. Update a bundle mod with a new bundle on the same path
3. Start the server from LAN IP (http.json, set host to `cmd > ipconfig` address, example: `192.168.178.32`)
4. Start the game from LAN IP
5. The bundle is redownloaded
### Remote, use cache
1. Ensure all bundles are cached
2. Start the server from LAN IP (http.json, set host to `cmd > ipconfig` address, example: `192.168.178.32`)
3. Start the game from LAN IP
4. The game starts loading bundles from the cache path
## Risk assessment
In order to reduce friction between releases, this PR introduces a deprecation system.
Obsolete classes and methods have been marked deprecated.
These will remain available for the current release and continue to function as normal for this release.
A warning will be displayed during build when a modder relies on the deprecated functionality.
The marked classes and methods are to be removed in the next release.
The server-side changes have no impact on modders.
## Deprecation
The following classes are affected:
- `Aki.Common.Http.Request`: Replaced by `Aki.Common.Http.Request`
- `Aki.Common.Http.WebConstants`: Replaced by functionality from `System.Net.Http`
- `Aki.Custom.Models.BundleInfo`: Replaced by `Aki.Custom.Models.BundleItem`
The following methods are affected:
- `Aki.Common.Http.RequestHandler.GetData(path, hasHost)`: `hasHost` enables connection outside intended host.
- `Aki.Common.Http.RequestHandler.GetJson(path, hasHost)`: `hasHost` enables connection outside intended host.
- `Aki.Common.Http.RequestHandler.PostJson(path, json, hasHost)`: `hasHost` enables connection outside intended host.
- `Aki.Common.Http.RequestHandler.PutJson(path, json, hasHost)`: `hasHost` enables connection outside intended host.
The deprecated methods and `Aki.Custom.Models.BundleInfo` are self-contained and can be removed independently.
The deprecated classes require removal of all deprecated code at once.
Reviewed-on: SPT-AKI/Modules#104
Reviewed-by: TheSparta <thesparta@noreply.dev.sp-tarkov.com>
Co-authored-by: Merijn Hendriks <merijn.d.hendriks@gmail.com>
Co-committed-by: Merijn Hendriks <merijn.d.hendriks@gmail.com>
## Preface
EFT has been reworking the `bsg.componentace.compression.libs.zlib` alot the past versions, including various enhancements and introduced bugs (zero-tail decompression infinite looping). This also includes working towards deprecating `SimpleZlib` and improving `ZOutputStream`'s performance.
While working on Haru, @waffle.lord and I have been reworking the zlib code to be much simpler. These changes are compatible with Aki.
## The issue
- The current code is complex to understand without experience with the zlib library in question
- `Zlib.IsCompressed` has a bug when operating on < 3 bytes. The third statement is reached and thus out of bounds.
## Why fix this
- Simplifying the code improves future readability.
- Using `ZOutputStream` enables usage of further performance improvements made to `ZOutputStream` by BSG.
- `Zlib.IsCompressed` will work again on `byte[2]` and below.
## Why was it like this in the first place?
At the time of writing this code, there was poor understanding of how the zlib library worked. The implementation was a best-guess from decompiled code.
## What's affected?
Only Zlib utility's internal code. No external libraries are affected.
Co-authored-by: Merijn Hendriks <merijn.d.hendriks@gmail.com>
Reviewed-on: SPT-AKI/Modules#43
Co-authored-by: Merijn Hendriks <senko-san@noreply.dev.sp-tarkov.com>
Co-committed-by: Merijn Hendriks <senko-san@noreply.dev.sp-tarkov.com>
## Preface
The current version of Aki uses .NET Framework 4.7.2, C# 7.3.
However, this is not correct. Unity 2019.4.31f uses .NET Framework 4.7.1, C# 7.0 (you can see this when creating a new blank project in this unity version, then check the Assembly-CSharp.csproj).
You can load netstandard2.0 assemblies into EFT, which support C# 7.3.
## The issue
.NET Core 2.1 support types like `Span<T>` and `ReadOnlySpan<T>` which previous versions of .NET (.NET Core 2.0 / .NET Framework 4.8.1 and below, C# 7.1 and below) need to support through the `System.Memory` nuget package. Right now, this conflicts.
If you attempt to load an assembly like this, EFT will simply crash because `Span<T>` and `ReadOnlySpan<T>` do not exist inside mono's `mscorlib`.
## Why fix this
This is important for a variety of reasons:
- Prevent modders from accessing not available APIs leading to unexpected crashes
- Prevent possible compatability issues with EFT later
- Support for `Span<T>` and `ReadOnlySpan<T>` through `System.Memory` which is important for high-performance code and client mods (one of which I'm working on).
I really want to stress that this is an important issue to fix and should be merged ASAP. Unity 2019.4 being lax doesn't mean the next LTS version won't prevent loading.
## Why was the wrong version in the first place?
It was a best guess made by me 2 years ago. There were more pressing issues back then and it didn't crash on me. It started becoming an issue 1 year ago when I tried to port Aki to C#, because self-signed certificate generation is supported only within 4.7.2 and newer.
For reference: https://learn.microsoft.com/en-us/dotnet/api/system.security.cryptography.x509certificates.certificaterequest.createselfsigned?view=netframework-4.7.2
## What's affected?
All the `.csproj` files in this repo.
Merging this means that some mods might need to retarget from .NET Framework 4.7.2 to 4.7.1.
Co-authored-by: Dev <dev@dev.sp-tarkov.com>
Co-authored-by: Terkoiz <terkoiz@spt.dev>
Co-authored-by: Terkoiz <terkoiz@noreply.dev.sp-tarkov.com>
Co-authored-by: DanW <danw@noreply.dev.sp-tarkov.com>
Co-authored-by: Merijn Hendriks <merijn.d.hendriks@gmail.com>
Reviewed-on: SPT-AKI/Modules#42
Co-authored-by: Merijn Hendriks <senko-san@noreply.dev.sp-tarkov.com>
Co-committed-by: Merijn Hendriks <senko-san@noreply.dev.sp-tarkov.com>
- Implement the models/helpers required to send console messages to the server console
- Implement sending plugin errors on startup to the server console
- Remove an unused variable and incorrect comment from PluginErrorNotifierPatch
Example output:
Depends on changes in SPT-AKI/Server#160
Co-authored-by: DrakiaXYZ <565558+TheDgtl@users.noreply.github.com>
Reviewed-on: SPT-AKI/Modules#35
Reviewed-by: Terkoiz <terkoiz@noreply.dev.sp-tarkov.com>
Co-authored-by: DrakiaXYZ <drakiaxyz@noreply.dev.sp-tarkov.com>
Co-committed-by: DrakiaXYZ <drakiaxyz@noreply.dev.sp-tarkov.com>
