Architecture & Internals

1  Pipeline & package layout

The extractor is a small Python package. Data flows in one direction — a read-only source in, decoders in the middle, JSON out — with no write path back to the game.

src/afkj_common/       shared by extract and (from P1b) parse; standard library only
  rules.py             locale codes, slugs and slug assignment, hero lookup, the season rule
  textkey.py           xxHash64 and text_key
  output.py            contained paths, staged writes, recoverable directory replacement
src/afkj_parse/        the parse capability (./afkj-parse; in progress, see the TDD)
  source/              one interface over extract output (primary) and the AFKJourneyData dump
  checks/              validation gates and out/build/report.{json,md}
  config.py, model/    config.json hero lists; the model layer
src/afkj_extract/
  safety.py            Source / LocalSource, discover_container(), is_game_running(), output_guard()
  session.py           lazy archive/runtime/Array and shared domain data for one run
  cdn.py               the game's CDN: package file table, verified range reads
  formats/
    binary.py          bounded spans, decompression limits, FormatError
    jsone.py           Blowfish + LZ4-block JSON frames
    pkg.py             igame.pkg TOC reader (+ per-entry .jsone)
    autogen.py         AutoGenConfig binary map tables (localization)
    luatable.py        Config/*.luae tables via a restricted Lua 5.4 VM + Array pool
    texture.py         AST/ASTC textures -> sRGB-encoded Pillow PNG (lazy heavy imports)
    luadis.py          Lua 5.4 bytecode constants and integer settings (mystical.codee, hero scripts)
  game.py              session-bound rules: locale file paths, hero rows, the current season, HeroScripts
  stages/              locales, tables, textures (+ --hero art filter), portraits,
                       artifacts, phantimals: one module per stage
  manifest.py          out/manifest.json and the raw stages' _stage.json stamps
  cli.py               argparse CLI, banner, safety gates, stage dispatch
tests/test_*.py        offline fixtures and regression tests
tests/smoke.py         optional live checks; --full compares every table cell
tools/                 investigation aids: unreleased skills, Lua disassembler (tools/README.md)
research/              manual investigations and image checks (images.html)

formats/* decode bytes and validate their structure. safety.py owns source paths; ExtractionSession shares one archive, runtime, Array pool, and frequently reused tables within a run. English strings are cached for naming; other language maps and skill partitions are emitted without retaining the full corpus. afkj_common holds the rules and output helpers extract shares with the parser (slugs, hero lookup, the season rule, text keys), game.py the ones that need a session, each module in stages/ produces one output folder, afkj_common.output owns publication, and cli.py handles arguments and diagnostics. This is a small local tool: ordinary functions and one session object are enough.

2  Read-only safety model

Game data is read through Source.read_bytes(), exists(), and glob(). There is no source-write API. Output protection also depends on explicit path checks; a read-only interface alone cannot prevent a separately chosen output path from aliasing the source.

• LocalSource rejects absolute paths, parent traversal, and symlinks that resolve outside its root.
• guards_output() rejects overlap in both directions. The output may neither sit inside the source nor contain it. The app bundle is also protected. Existing symlink components and hard-linked output files are refused by destination checks.
• Container discovery reads the bundle ID from container metadata. If permissions prevent discovery, supply an explicit --source /path/to/data or a local copy.
• The process guard (safety.require_game_closed) lists processes through libproc (under a millisecond; pgrep -x AFKJourney if libproc is unavailable) and runs before every access to a path under the app bundle or ~/Library/Containers: reads, existence checks, listings, even resolving a path, and the container discovery scan. An open game or a failed check refuses the access; there is no override, dry runs included. The research scripts that open game paths directly call it before each file. A --source copy outside those trees is not a game file and is not checked.
• Each stage is built beside its destination, then the old directory is backed up and the new one renamed into place. An ordinary publication failure restores the backup. If restoration also fails, the backup path is reported and retained. Filtered runs start with a copy of the previous output; full runs remove stale files.

This protects normal single-user operations, not concurrent hostile filesystem changes or arbitrary untrusted Lua bytecode. Publication is per stage, not a transaction across the whole invocation. Avoid concurrent runs targeting the same output directory.

3  Data sources

The extractor auto-discovers the container; --source can point it at a manual copy (e.g. a dump on another disk) instead.

Languages. A language file (Config/AutoGenConfigBytes/LanguageDb<CODE>.bytes) only exists locally once that language has been selected in-game, so a typical install has just a couple. The extractor emits every one it finds (all 17 possible) and the parser filters: the feed leaves out cn as a near-duplicate of zh (252,529/252,543 strings identical) but still reads it internally, and its in-game selector shows 16 because cn is the hidden Simplified source variant. afkj-extract --list prints which locales are present vs. missing so you know what to download in-game; --locales-from cdn fetches all 17 from the CDN instead (integrations §1).

Updates and patches. The loose files are patched in place, so what the extractor reads is always the current resource version. For reference, in Data/Documents/ (one level above data/):

• FullFileHash.iOS.<version>.bin is written once per resource version (1410363 as of 2026-09-17); older ones are left behind.
• afk.patch.bin is LZ4 (u32 size prefix) around afk.patch.pkg.bin, a list of binary diffs named <from>-<to>.<path>.hdiff, e.g. 1314627-1410363.Config/AutoGenConfigBytes/LanguageDbEN.bytes.hdiff. afk.bin (also LZ4) and afk.pkg.bin are file manifests (see the CDN callout).
• data/InjectPatch/*.patch.bytes are compiled C# hotfixes (IFix) for Assembly-CSharp, not data or text.
• data/patch/<version>/multilanguage/ and data/multilanguage/{cn,en} exist but are empty. Config/LanguageDb{EN,CN}.luae (February) are older Lua-format language tables that the .bytes files superseded.

3.1 What's on disk (packages and file kinds)

A census of both locations, 1.6.11 (2026-09-18), from research/package_inventory.py (12 s; it walks .lpak headers without decompressing). Container: 33,308 files, 16.9 GB. App bundle: 429 files, 1.37 GB. "Bundles" counts the UnityFS bundles inside the packs (images §3). Where the app and the container have the same file, the container's is newer and is the one the game uses.

The app bundle's Data/Raw/ holds igame.pkg, one AutoGenConfig file, 45 preinstall ui/icon images, 85 sound banks, a movie, and aa/iOS/ with 280 packs (0.96 GB, 20,519 bundles: 276 hashed packs, the three pre_install_* packs and builtin). Its pre_install_unmanaged_assets is an older build of the container's (141 vs 132 bundles, a different Common atlas).

4  Format: .jsone verified

Blowfish-encrypted, LZ4-compressed JSON. Used for small loose config (game version, etc.). formats/jsone.py:

offset 0 : magic  FF FF FF FF
offset 4 : declaredSize u32 LE
offset 8+: Blowfish-ECB(key="b8c0aeff2944d1614d6bc63dc6b0a537", 8-byte blocks)
           with a per-block half-swap: reverse each 4-byte half BEFORE and AFTER
           the cipher call (reconciles the impl's word order with LE data)

decrypted payload:
  [prefixLength u32][prefix][decompressedSize i32][LZ4-block data]  ->  UTF-8 JSON

API: jsone.decode(bytes) -> bytes, jsone.decode_json(bytes) -> obj.

5  Format: .pkg verified

A custom indexed container (not a ZIP). Two files work together:

• igame.pkg.bin — an LZ4-block-compressed table of contents: (u32 nameLen, name, u32 size, u32 offset) per entry (~10,155 entries).
• igame.pkg — the payload; each TOC offset points at a standalone .jsone frame.

Entries hold compiled Lua (Src/*.luae game logic and, crucially, Config/*.luae data tables) plus the protobuf schema Pb/igame.pbe. API: PkgArchive(pkg, toc).read(name) -> bytes (decodes the entry's .jsone frame), .names(), name in archive.

6  Format: AutoGenConfig .bytes verified

The game's binary map format, used for localization (LanguageDb{EN,CN}.bytes) and index tables. A hash-map serialized as four regions:

header (16 bytes): u32 version=0, u32 headerSize=16, u32 entryCount, u32 slotCount
region A: slotCount  x 16-byte hash slots         (lookup index — unused for a dump)
region B: entryCount x 16-byte records            (u64 key, u32 valuePtr, u32 chain)
region C: value pool

string value @ valuePtr:  [u16 len LE][utf-8 bytes]

To dump, iterate region B (offset headerSize + slotCount*16) reading each (key, valuePtr) and decoding the value. autogen.decode_string_map(bytes) -> {key: text} handles the string case, which is all localization needs. (Typed/int columns use a tagged word encoding that isn't reversed; moot for the parser, since the tables it needs come from Lua, below.)

Keys are xxHash64 of the Chinese source text. Every localization key is xxHash64 (seed 0) of the UTF-8 Simplified Chinese string it translates (afkj_common.textkey.text_key, checked on 3,000 of 3,000 entries). Tables in this build store that Chinese text raw (Unit.DisplayName, skill Description/SimpleDescription), so any language resolves as LanguageDb<LANG>[text_key(field)]; an all-digit field is already a key. The Chinese locale file does not list every source string, so the key has to be computed rather than looked up.

Some keys ship blank. A key can be present with an empty value in every language. Karma's Ultimate Lv1 description (key 15602405291547514597) is one: the game shows its English in-game, but no file on disk holds it. The search covered both language files (current and the older .luae), every decoded igame.pkg entry, every AutoGenConfig table, the HTTP caches and stores (Library/Caches, igame.kv, *_expedite.dat, including gzip/zlib), and every TextAsset/MonoBehaviour in the Unity bundles: all 6,237 container packs, split into their 231,342 bundles (1.6 M objects, research/bundle_text_search.py; a first pass had opened each pack whole and seen only its first bundle). The guess then was runtime delivery; in fact the text shipped late: build 14368 (2026-09-25) carries it under the same key in every language. The in-game text also added "+ Ult Power" terms (UAP) that no template contains, so the UI appends those itself. When a description is blank, take it from an in-game screenshot until a later build fills it. Check first that the game shows the skill at all: a row with IsShow false is an internal sub-skill (Raw data §9.2).

7  Format: Config/*.luae tables verified

The centerpiece. Every hero/skill table is a compiled Lua 5.4 data module inside igame.pkg — Config/Unit.luae, Config/SkillBuff.luae, Config/TavernPool.luae, Config/Skill/Skill_*.luae, and so on. They are standard Lua 5.4 bytecode (no custom opcodes), which is what makes the following possible.

7.1 Restricted execution of trusted game data

luatable.py runs stock Lua 5.4 bytecode through lupa. Each module receives fresh copies of the table, string, and math library tables, a small pure base API, and the game's pure isnumber/table.bseek helpers. File/process libraries and the Python bridge are not exposed in the module environment. Unknown globals fail with their names. The shared Array pool is deliberate; other module configuration is fresh. This in-process API restriction is intended for trusted local game files and is not a security sandbox for hostile bytecode.

7.2 What a module builds

Running a module assigns two things into config:

config.<Name>HeaderTable = { field -> columnIndex }        -- e.g. { ID=0, Name=1, HP=26, ... }
config.<Name>Table       = <accessor>                       -- lazy row view over a flat cell array

accessor.__dataTable = { [1]=flatCells, [2]=rowKeys, Len=cellCount }

flatCells is a sparse array of rowCount × stride cells (a missing cell means "use the column default"); stride = max(columnIndex) + 1. Row r, column c lives at flatCells[r*stride + c + 1]. Skill partitions nest one level deeper (config.Skill.Skill_0Table); find_tables() handles both.

7.3 Exact schema recovery

Defaults and array-column indices are captured by the row materializer. The extractor follows the accessor metatable one level for ordinary tables or two for grouped tables, then matches the captured header and data by Lua identity. Supported scalar and array materializers have three and eight upvalues respectively. Unknown shapes fail explicitly. Header indices may be noncontiguous; stride remains max(columnIndex) + 1. Missing defaults become null, matching Lua nil, and empty array defaults become []. No type-based default guessing is used.

7.4 Array resolution

Config/Array.luae stores flat values in A[1] and one-based start offsets with a final sentinel in A[2]. Reference N is A[1][A[2][N] .. A[2][N+1]-1]. Numeric indexing preserves nil holes as JSON null; iteration through .values() would lose their positions. Offsets and references are validated. For game 1.6.11, reference 110932 is [null, "false", "true"], and 110933 is ["atk", "atk6", "atkb"], verified against the game's own accessors.

7.5 Reconstruction and grouped keys

For each row, absent cells use the recovered default, numeric array cells resolve through the pool, inline tables become lists/maps, and scalar cells retain their values. Grouped tables store group/size pairs in __dataTable[3]. Their inner keys repeat across groups, so rows use JSON-encoded composite keys such as "[1,1]", with keyFields and rowKeyFormat: "json-array" metadata. This prevents overwriting earlier groups. Single-key table keys are unchanged.

SkillIndex.luae is a plain ID-to-partition map, exported separately as {fields: ["ID", "Partition"], arrayFields: [], rows: {...}}.

8  Format: AST textures verified

The game's loose images (under ui/, ~11,800 of them) are ASTC textures in a thin wrapper — named .png but not actually PNG. formats/texture.py:

offset 0  : magic  "AST"
offset 3  : width  u16 LE
offset 5  : height u16 LE
offset 7  : format u8       (ASTC block size: 0x0d = 6x6, 0x0e = 8x8)
offset 8  : decompressedSize i32 LE
offset 12 : LZ4-block-compressed ASTC blocks

decode: LZ4 decompress -> texture2ddecoder.decode_astc(w,h,bw,bh) -> BGRA
        -> flip vertically (GPU origin is bottom-left) -> Pillow RGBA
        -> sRGB-encode RGB (texels are linear; skip with --linear) -> PNG

The block-size map is verified against the whole corpus (ceil(w/bw)·ceil(h/bh)·16 == decompressedSize for every file). Heavy deps (Pillow, texture2ddecoder — the [textures] extra) are imported lazily so the rest of the package doesn't require them. The textures stage is opt-in and mirrors the ui/ tree into out/raw/textures/; a --textures GLOB filter subsets the ~12k images, and --hero selects one hero's art (below).

Colour handling, finding a hero's art, art in Unity bundles and UI sprites are in Images (§1–4), with the portrait, artifact and Phantimal exports (§5–6).

9  Output format

The output shape is ours to choose, since afkj-parse is its reader, so it favors being directly consumable over mirroring AFKJourneyData's _meta/$arrayRef structure.

out/
  manifest.json                 { schemaVersion: 2, tool, toolVersion, gameVersion, generatedAt,
                                  source, scope, status, options, stats, error? }
  raw/locales/en.json           { "<key>": "<text>" }        (301k strings; runtime key resolution)
  raw/locales/_stage.json       { formatVersion, extractor, gameVersion, fileTable, from: install | cdn, cdnBuild?, locales }
  raw/tables/Unit.json          { fields:[...], arrayFields:[...], rows: { "<key>": { ...full row... } } }
  raw/tables/Skill/Skill_0.json per-skill partitions
  raw/tables/HeroScripts.json   { Script, Name, Value } rows: hero battle script settings (raw data §10.5)
  raw/tables/_stage.json        { formatVersion, extractor, gameVersion, fileTable, complete }
  raw/cdn/<build>/…             the CDN download cache (--locales-from cdn)
  raw/textures/**/*.png         decoded AST/ASTC art (only when the textures stage runs)
  images/heroes/<slug>.png      hero roster portraits (portraits stage, images §5)
  images/heroes/alt/, manifest.csv   skin alternates, provenance, in_reference
  images/artifacts/<slug>.png   permanent and current-season artifacts; manifest.csv `set` (images §6)
  images/phantimals/<slug>.png  the current season's Phantimals (images §6.5)
  tools/                        outputs of tools/

raw/ keeps the game's own names so a finding can be traced to its source; images/ uses the slugs the feed uses (afkj_common.rules.assign_slugs), so a client joins a feed entry to its image by slug. Each raw stage stamps its own directory: fileTable is xxHash64 of the install's afk.bin, and the CDN option refuses to run unless the CDN's file table hashes the same, so the stamp proves text and tables are one build. formatVersion is the contract with the parser; complete is false after a filtered run.

Table rows are full objects (every field present, defaults applied, arrays resolved inline), keyed by primary key or the composite grouped key described above. Localization stays a flat {key: text} dict per language — the form the viewer already resolves at runtime for locale switching; inlining strings into rows would multiply size across 17 locales and break that mechanism.

manifest.json describes the last invocation: selected stages/tables/heroes, limits, color and reference options, completion status and statistics. Unselected outputs can be older. Preflight failures and dry runs leave the previous manifest untouched. Full stages replace stale files; filtered/limited stages preserve unselected files. Runtime failures record a failed manifest when the destination remains writable.

10  Verification

Use make check for Ruff and the offline regression suite, and make build for wheel/source packaging. Generated fixtures exercise binary truncation and bounds, schema/default recovery, array holes, grouped keys, path aliases, publication failures, filtered runs and CLI errors without game files.

• --verify <ref> compares missing, extra and changed locale keys. Differences return a failure status; version skew can explain a difference but is not assumed to do so.
• PYTHONPATH=src python3 tests/smoke.py --source /path/to/data --full runs read-only live checks and compares every reconstructed table cell with the real Lua accessors. The September 20, 2026 validation on game 1.6.11 matched 875 accessor-backed tables, 17,157 rows and 1,846,424 cells. SkillIndex is checked separately in the offline suite and included in the 876 exported table files.
• The refactor retained byte-identical output for all 186 portrait/alternate PNGs and 18 artifact PNGs when compared with the pre-refactor implementation.
• The 2026-09-26 restructure (emit.py into game.py + stages/, scratch/ into tools/ + research/) kept every output byte-identical to the previous code on the same install: all six stages (1,200 files, textures limited to *herohead*), the manifest's stats, the then viewer bridge (941 files), artifact_stats.py and unreleased_skills.py. smoke.py --full on build 14368 then matched 918 tables, 17,738 rows and 1,913,090 cells.
• Tools (tools/) are linted by make check, and tests/test_cdn.py pins the CDN file-table parser.
• Image checks live in research/: color_screencheck.py (in-game screenshots), srgb_equivalence.py (encode vs URP/ICC), portrait_gpu_check.py + gpudecode.swift (GPU ground truth), the replacement checks in images §5.2 and §6.2 (the same tools cover portraits and artifacts), and atlas_sprites.py (finds and dumps UI sprites in the bundles, images §3).
• Game files: research/package_inventory.py (the census in section 3.1), bundle_text_search.py (text in every bundle, section 6).
• Image files: research/png_audit.py (what a PNG carries besides pixels, noise vs colour tags; images §6.4).

11  Design history

The design review that started this project (afkj-extractor-review.html, in git history) was retired on 2026-09-25; its findings live in the sections above. What it recorded that isn't stated elsewhere:

• Why the original notes misled. They followed AnimeStudio's AFKJourneyUtils.cs, which targets the Windows build. The iOS-on-Mac build differs: .pkg is a custom TOC plus .jsone frames, not ZIP (section 5); textures are AST/ASTC, not DXT/BC7 (section 8); and the hero and skill tables are compiled Lua in igame.pkg, not in the Unity bundles or the loose .jsone files (section 7). The container is UUID-named, found by bundle id (section 3).
• Why Python. The one hard dependency was reading the Unity .lpak bundles, where UnityPy is the mature option; every other decoder was already proven in Python. It is a hand-run batch tool, so speed and distribution didn't weigh.
• Why our own output shape. With the extractor and the parser both ours, the extractor resolves defaults, arrays and partitions at the source (section 9), and AFKJourneyData became a cross-check and the fallback source. afkj-parse reads either layout (Raw data §2).
• A corrected early claim. The review read Config/Array's offsets as cumulative ends and "verified" it on common_taunt. They are starts with a sentinel (section 7), and the viewer had the same misreading (Raw data §6).

11.1 Parse design decisions

These are from the parse-capability plan (2026-09-25 to 27), retired when the work finished. Each row gives the choice, what was rejected and why, so a change can be weighed against the reason. A later row carries its own date.

The rule behind several of these, and the one to apply to new cases: take the game's own data over rules kept by hand.