diff --git a/docs/dev/architecture.md b/docs/dev/architecture.md index c79bec1..7c36e4c 100644 --- a/docs/dev/architecture.md +++ b/docs/dev/architecture.md @@ -1 +1,160 @@ # Architecture + +This page describes how `foreignthon-core` works internally. + +--- + +## Pipeline + +``` +source.xx.py + │ + ▼ +_check_shebang() ← reads "# foreignthon: xx" if present + │ + ▼ +load_pack(lang_code) ← discovers + validates the JSON pack + │ + ▼ +_apply_postfix_syntax() ← rewrites "expr @@keyword:" lines + │ + ▼ +_swap_tokens() ← tokenizer pass: replaces NAME tokens + │ + ▼ +standard Python string ← ready to compile or write to disk +``` + +--- + +## Module overview + +| Module | Responsibility | +|---|---| +| `transpiler.py` | The engine — postfix rewriter and tokenizer pass | +| `pack.py` | Pack discovery, loading, and validation | +| `cli.py` | Click commands (`fpy run`, `fpy compile`, etc.) | +| `errors.py` | Bilingual exception hook | +| `template.json` | Canonical set of all keywords/builtins a pack must cover | + +--- + +## Tokenizer-based translation + +ForeignThon uses Python's standard `tokenize` module rather than regex or AST manipulation. + +`tokenize.generate_tokens()` splits source code into typed tokens. ForeignThon only looks at `NAME` tokens — identifiers. It replaces any `NAME` token whose string appears as a key in the active pack mapping. All other token types (strings, comments, operators, numbers) pass through unchanged. + +This gives three important guarantees: + +1. **Strings are safe.** A keyword inside `"..."` or `f"..."` is a `STRING` token, never a `NAME` — it is never touched. +2. **Comments are safe.** Comment tokens are passed through verbatim. +3. **Variable names are safe.** A variable like `si_condition` contains `si` only as a substring; as a `NAME` token it is `si_condition`, which is not in the mapping. + +The whitespace between tokens is preserved by tracking `(row, col)` positions and copying the gaps from the original source. + +--- + +## Pack discovery + +Language packs register themselves using Python [entry points](https://packaging.python.org/en/latest/specifications/entry-points/): + +```toml +# in foreignthon-es/pyproject.toml +[project.entry-points."foreignthon.langs"] +es = "foreignthon_es" +``` + +`pack.py` calls `importlib.metadata.entry_points(group="foreignthon.langs")` at runtime to discover all installed packs. Installing a pack is sufficient — no configuration file needs to be edited. + +Each pack module must expose: + +```python +def get_pack_path() -> Path: + return files(__name__) / "xx.json" +``` + +The core calls `get_pack_path()` to locate the JSON, loads it, and validates that all required sections are present. + +Results are cached with `@lru_cache` so each pack is loaded at most once per process. + +--- + +## Pack mapping + +Four sections of the JSON are merged into a single flat dict for translation: + +```python +mapping = {} +mapping.update(pack["keywords"]) +mapping.update(pack["builtins"]) +mapping.update(pack["exceptions"]) +mapping.update(pack["stdlib"]) +``` + +The merged mapping is `{ foreign_word: english_word }`. It is passed directly to `_swap_tokens()`. + +If two sections define the same foreign key, later sections win (stdlib last). In practice this does not occur because pack authors ensure uniqueness. + +--- + +## Postfix syntax (`@@`) + +The `@@` operator is a source-level pre-processing step that runs **before** tokenization. + +A line like: + +``` +x > 0 @@si: + escribir(x) +``` + +is rewritten to: + +``` +si x > 0: + escribir(x) +``` + +The rewriter uses a regex that matches `(.+?)@@()` and moves the keyword to the front. It only operates on lines that contain `@@`, preserving indentation and line endings. + +`@@` is never valid Python and never appears in the tokenizer output. + +**Decompile direction:** `fpy decompile --postfix` does the reverse — it looks for lines of the form `foreign_kw expr:` where `foreign_kw` is in the pack's `postfix_keywords` list, and rewrites them to `expr @@foreign_kw:`. + +--- + +## Bilingual error hook + +`errors.py` installs a custom `sys.excepthook` before running user code: + +1. On exception, it looks up the exception type name in the pack's `exceptions` section (reverse map: English → foreign). +2. It looks up a translated message in `error_messages`. +3. It prints `[XX] ForeignName: translated_msg` then `[EN] EnglishName: original_msg`. +4. It always calls `traceback.print_exception()` afterwards so the full traceback is shown. + +Tracebacks point to the original `.xx.py` file. This is achieved by populating `linecache.cache` with the original source before `exec()`-ing the compiled code, so Python's traceback machinery reads the right lines. + +--- + +## Custom pack override + +When `.foreignthon.toml` declares `custom_pack = "path/to/custom.json"`: + +- If the custom JSON has `meta.code` set, it is treated as a **standalone pack** and used directly. +- If `meta.code` is absent, it is treated as an **override** — it is merged on top of the installed pack, replacing only the keys it defines. + +The CLI (`cli.py`) handles this in `_load_effective_pack()` by walking up the directory tree to find `.foreignthon.toml`. + +--- + +## File naming and language detection + +Language detection order (highest priority first): + +1. `--lang` CLI flag +2. Shebang comment `# foreignthon: xx` on the first line +3. Double extension `.xx.py` → `xx` +4. Fallback to `"en"` (no-op — English is Python) + +`_detect_lang()` and `_check_shebang()` in `transpiler.py` implement steps 3 and 2 respectively. Step 1 is handled by the `run` command in `cli.py`. diff --git a/docs/dev/releasing.md b/docs/dev/releasing.md index af81e9e..de28ea2 100644 --- a/docs/dev/releasing.md +++ b/docs/dev/releasing.md @@ -1 +1,136 @@ # Releasing + +This page covers how to release `foreignthon-core` and how language pack maintainers should respond to a new core release. + +--- + +## Core release (`foreignthon`) + +### 1. Update the version + +Version is set in `pyproject.toml`: + +```toml +[project] +name = "foreignthon" +version = "0.5.4" # bump this +``` + +ForeignThon follows [Semantic Versioning](https://semver.org/): + +| Change | Version bump | +|---|---| +| Bug fix, no API change | Patch (`0.5.3` → `0.5.4`) | +| New feature, backward compatible | Minor (`0.5.x` → `0.6.0`) | +| Breaking change (removes a key, changes CLI contract) | Major (`0.x.y` → `1.0.0`) | + +### 2. Update `template.json` if needed + +If the release adds new keywords, builtins, or sections, update `template.json` to include them. This is the canonical list all language packs are expected to cover. Document what changed in your commit message and PR so pack maintainers know what to add. + +### 3. Run CI locally + +```bash +pytest +ruff check src/ +ruff format --check src/ +``` + +All must pass. + +### 4. Tag and push + +The Gitea CI publishes to PyPI on any tag matching `v*`: + +```bash +git tag v0.5.4 +git push origin v0.5.4 +``` + +The `publish.yml` workflow builds the wheel and uploads it using the `PYPI_TOKEN` secret configured in Gitea. + +### 5. Verify + +```bash +pip install --upgrade foreignthon +fpy --version +``` + +--- + +## Language pack release (`foreignthon-xx`) + +Each language pack is independently versioned and released. + +### Responding to a core release + +When `foreignthon-core` adds new entries to `template.json`: + +1. Add the new keys to your `xx.json` with translations. +2. Run `fpy pack src/foreignthon_xx/xx.json` to validate. +3. Run `pytest tests/` to confirm all checks pass. +4. Bump your pack's version in `pyproject.toml`. +5. Tag and push — the `publish.yml` workflow handles PyPI upload. + +Update the `foreignthon` dependency lower bound in `pyproject.toml` if your pack requires the new core version: + +```toml +dependencies = ["foreignthon>=0.5.4"] +``` + +### Releasing independently + +You can release a pack at any time — to fix a translation, add missing entries, or extend coverage. The process is the same: + +```bash +# bump version in pyproject.toml +git add pyproject.toml src/foreignthon_xx/xx.json +git commit -m "v0.3.3: fix translation for 'return'" +git tag v0.3.3 +git push origin v0.3.3 +``` + +--- + +## Docs release (`foreignthon-docs`) + +The docs site rebuilds automatically when a language pack fires its `trigger-docs.yml` workflow after a successful publish. No manual step is needed for routine pack releases. + +To manually trigger a docs rebuild (e.g. after editing `foreignthon-docs` directly): + +```bash +git push origin main +``` + +The `deploy.yml` workflow builds the MkDocs site and deploys it to [foreignthon.keshavanand.net](https://foreignthon.keshavanand.net). + +--- + +## Gitea secrets + +| Secret | Used by | +|---|---| +| `PYPI_TOKEN` | `publish.yml` in core and all packs | +| `DOCS_WEBHOOK` | `trigger-docs.yml` in language packs | + +Secrets are set per-repository in **Gitea → Settings → Actions → Secrets**. + +--- + +## Checklist + +### Core +- [ ] Version bumped in `pyproject.toml` +- [ ] `template.json` updated if new keys added +- [ ] `pytest` passes +- [ ] `ruff check` passes +- [ ] Tagged `v*` and pushed +- [ ] PyPI upload confirmed + +### Language pack +- [ ] New `template.json` entries translated in `xx.json` +- [ ] `fpy pack xx.json` passes +- [ ] `pytest tests/` passes +- [ ] Version bumped +- [ ] `foreignthon` dependency lower bound updated if required +- [ ] Tagged and pushed