EFT internals research: File integrity scanning-ODDBA社区

SPT主创

初窥堂奥

VIP2

SPTarkov项目组

Hello everyone!

Today, I would like to cover 4 aspects of EFT's internals in 4
different articles: secured comminucation, payload encryption, response
caching and file integrity scanning. These articles are mostly aimed at
SuperMod author and anyone else interested in making modding more
secure.

In this last article, I'll be talking about another completely unexplored territory; file integrity validation.

File integrity validation

Ever had the game instantly close on you when you start the game the wrong way? If so, that was file integrity kicking in and detecting you run AKI.

In EFT, there are two validation checks: one by BsgLauncher, another by the game itself (introduced in 0.12.11). BsgLauncher uses the info stored in a file called ConsistencyInfo. The game on the other hand has the info hard-coded inside FilesChecker.dll. By having these two checks, it detected early on alot of cheaters who made use of simple timely injection scripts. These days the tricks are long bypassed.

AKI opts for completely disabling the checks by making them always return true. My research server on the other hand reimplements the checks so the server can do custom verification.

Here is what the data structure roughtly looks like:

    [DataContract]
    public struct ConsistencyMetadata
    {
        [DataMember]
        public string Path;
        [DataMember]
        public long Size;
        [DataMember(EmitDefaultValue = false)]
        public string Hash;
        [DataMember(EmitDefaultValue = false)]
        public int? Checksum;
        [DataMember(EmitDefaultValue = false)]
        public bool? IsCritical;
    }

There is a slight problem; BsgLauncher and the game itself use two different methods for validation. The launcher checks a MD5 hash, whereas the game uses a checksum. because of this, I opted for implementing both in case the need arises for supporting both.

Some files (like Assembly-CSharp.dll) might be marked as critical. Non-critical files are only checked for if they exist and if the filesize is correct. Critical files however also validated for the hash/checksum. While implementing these checks itself is not difficult, the real challenge is to make it performant. To get a rough idea how slow critically-checking all files is: 6200
files (some of which 200+ MB) take 1 minute on a 6-core laptop processor
with multi-threaded checking.

In dotnet 6 (A language runtime for C#), we can use Parallel.ForEachAsync() to run the code multi-threaded. However, this is not available in EFT's C# runtime. My only option here is to re-implement the functionality myself. Welcome to my own thread scheduler!

        // https://github.com/dotnet/runtime/blob/main/src/libraries/System.Threading.Tasks.Parallel/src/System/Threading/Tasks/Parallel.ForEachAsync.cs
        public static Task ForEachAsync<T>(
            IEnumerable<T> source,
            Func<T, CancellationToken, Task> body)
        {
            if (source == null)
            {
                throw new ArgumentNullException(“source”);
            }
            if (body == null)
            {
                throw new ArgumentNullException(“body”);
            }
            if (Options.MaxDegreeOfParallelism < 1)
            {
                throw new IndexOutOfRangeException(“Options.MaxDegreeOfParallelism”);
            }
            var threadCount = Options.MaxDegreeOfParallelism;
            var threads = new Task[threadCount];
            var scheduler = Options.TaskScheduler ?? TaskScheduler.Current;
            var ct = Options.CancellationToken;
            var cts = CancellationTokenSource.CreateLinkedTokenSource(ct);
            var enumerator = source.GetEnumerator();
            var semaphore = new SemaphoreSlim(1, 1); 
            for (int i = 0; i < threadCount; ++i)
            {
                threads[i] = Task.Factory.StartNew(async () =>
                {
                    try
                    {
                        while (true)
                        {
                            if (cts.IsCancellationRequested)
                            {
                                ct.ThrowIfCancellationRequested();
                                break;
                            }
                            // continue on captured context.
                            await semaphore.WaitAsync();
                            T item;
                            try
                            {
                                if (!enumerator.MoveNext())
                                {
                                    break;
                                }
                                item = enumerator.Current;
                            }
                            finally
                            {
                                semaphore.Release();
                            } 
                            // continue on captured context.
                            await body(item, cts.Token);
                        }
                    }
                    catch
                    {
                        cts.Cancel();
                        throw;
                    }
                }, CancellationToken.None, TaskCreationOptions.DenyChildAttach, scheduler).Unwrap();
            }
            var continuationOptions = TaskContinuationOptions.DenyChildAttach
                                    | TaskContinuationOptions.ExecuteSynchronously;
            return Task.WhenAll(threads).ContinueWith(t =>
            {
                // Clean up (dispose all disposables)
                enumerator.Dispose();
                cts.Dispose();
                semaphore.Dispose();
                return t;
            }, CancellationToken.None, continuationOptions, TaskScheduler.Default).Unwrap();
        }

This is all really complicated, so in short: I grab the threads (except the primary thread) available, then offload tasks (things to do) onto them. In between tasks I need to synchronize, which the semaphore is used for.

There is a nice benefit to this implementation; while Parallel.ForEachAsync adds threads to use over time, my implementation immediately uses all threads. This makes the code more performant in scenarios like this. The result of this is much faster file validation than EFT currently provides.

        public async Task<Dictionary<string, ConsistencyResult>> ValidateFiles(
            string basepath,
            ConsistencyMetadata[] metadatas)
        {
            var results = new ConcurrentDictionary<string, ConsistencyResult>();
            await Scheduler.ForEachAsync(metadatas, async (metadata, ct) =>
            {
                var result = await ValidateFile(basepath, metadata);
                results.TryAdd(metadata.Path, result);
            });
            return new Dictionary<string, ConsistencyResult>(results);
        }

This is especially nice for bundle loading. With a bit of code reworking, it would be possible to run most of the bundle loading code on separate threads, instead of asynchroniously on the same thread, resulting in faster loading of the game.

How do modders benefit from this?

With this implementation, it is now possible to have server-side validation of your files. If a person tampered with your mod, you can now prevent loading of the game and warn the user not to do it or to redownload the mod. One more protection layer for modders to use!

Do only modders benefit from this?

No, normal users benefit too. Since the code is faster to run, the client will load faster too. And you now got a way to validate that your client is set-up correctly, making it easier to diagnose issues.