1. What is a data pipeline?

Plain English: A data pipeline is a series of steps that data flows through, one stage at a time, from raw input to a useful result. Each stage does one job and hands its output to the next stage.

Most serious projects follow the same staged roadmap: preserve the raw data → build a clean, canonical layer → build a search layer → evaluate. Why this order? Because each stage protects the next. If you keep the raw data safe, you can always rebuild everything else. If your clean layer is trustworthy, your search results will be trustworthy too.

2. What is XML, and why can’t we search a huge XML file directly?

Plain English: XML (eXtensible Markup Language) is a text format that wraps every piece of data in labelled tags, like <title>Deep Learning</title>. It is easy for both people and computers to read.

Our example project starts from the DBLP computer-science bibliography, which publishes its entire database as one giant XML file (the monthly “Snapshot XML Release” from the dblp Team at Schloss Dagstuhl). One measured version, the 2021 release with about 6.15 million records, arrived as a 611 MB compressed archive that unpacked into a single dblp.xml file of 3.2 GB — and the file keeps growing every month. By 2026 the compressed download is around 1 GB, and dblp’s own parsing FAQ notes the uncompressed file has “grown to no longer fit into the standard 4GB memory” and recommends giving a parser 8 GB.

A file that size cannot fit comfortably in a computer’s memory all at once, and it has no built-in way to jump to one record. This is why we use streaming parsing: instead of loading the whole file, the program reads it one record at a time, does something with that record, and then throws it away before reading the next. Memory use stays small and steady no matter how big the file is. (dblp itself advises using a streaming approach like SAX or StAX rather than loading the whole document.)

3. What is SQLite? Tables, schemas, migrations and backups

Plain English: SQLite is a database that lives inside a single file on your disk. There is no server to install or start — your program opens the file and reads or writes directly.

Inside a database, data lives in tables. A table is a grid: each row is one item (one paper), and each column is one field (title, year, authors). The schema is the blueprint that says which tables and columns exist and what type of data each column holds.

Projects change over time, so the schema must change too. A database migration is a small, numbered SQL file that makes one change — for example, adding a missing column. Numbering them (001_init.sql, 002_add_venue.sql) keeps every copy of the database in the same known state.

-- 002_add_venue.sql
ALTER TABLE papers ADD COLUMN venue TEXT;

Two safety habits matter. Read-only protection means opening the raw database in a mode where writing is blocked, so you cannot accidentally change your original data. Backups mean keeping copies — and because a SQLite database is just one file, backing it up is as easy as copying that file.

4. What does “source-preserving” storage mean?

Plain English: Source-preserving storage means you keep every original value exactly as it arrived — same spelling, same spacing, same everything — and never rewrite it.

5. What is canonical mapping (and what is provenance)?

Plain English: Canonical mapping means translating each source’s private field names into one shared, neutral set of field names that every source agrees on.

Different data sources speak different “dialects”. DBLP calls a conference name booktitle and a journal name journal. Another source such as PubMed uses its own different labels. A canonical format is one common language — for example title, authors, year, venue, and identifiers — that every source maps into.

Why bother? Because once every source speaks the canonical language, you can federate them — search DBLP, PubMed and others together as if they were one dataset. A strict rule keeps this honest: never guess missing values. If a source has no year, the canonical year stays empty. Inventing data would poison every later step.

Provenance is the receipt for every value: a record of where it came from. For each canonical field you store which source and which original field produced it, so you can always trace a value back to its origin.

6. What is automated testing, and what is pytest?

Plain English: Automated testing means writing small programs that check your main program behaves correctly. You run them anytime, and they tell you pass or fail in seconds.

pytest is the leading testing framework for Python; in the Python Developers Survey it is used by almost half of all Python users, ahead of the built-in unittest. A unit test checks one small piece (“unit”) of your code in isolation — for example, one function.

def canonical_year(record):
    return record.get("year")

def test_missing_year_stays_empty():
    assert canonical_year({}) is None

Test-driven development (TDD) means writing the test first, watching it fail, then writing just enough code to make it pass — the “red, green, refactor” cycle popularised by Kent Beck. This forces you to decide exactly what “correct” means before you build. When people say “100 tests green”, they mean all 100 checks passed — a quick, reassuring sign that recent changes did not break anything.

7. What is Ruff, and what is a linter?

Plain English: A linter is a tool that reads your code and points out style problems and likely bugs — without running the code.

Ruff is a Python linter and code formatter made by the company Astral and written in the Rust programming language. First released in late 2022 by Charlie Marsh, it is described in its own docs as “10–100x faster than existing linters (like Flake8) and formatters (like Black)” — one Bokeh co-creator reported it scanning a whole repository “in ~0.2s instead of ~20s.” It topped the Stack Overflow Developer Survey 2025 as the most-admired developer tool and is used by projects like FastAPI, Pydantic, Pandas and Hugging Face Transformers.

8. What is mypy, and what are type hints?

Plain English: Type hints are small notes in your code that say what kind of value something should be — text, a whole number, a list, and so on. mypy is a tool that reads those hints and checks you are using them consistently, without running the program.

Here is a tiny bug that mypy catches before the program ever runs:

# Before: no hints, hidden bug
def add_year(year):
    return year + 1

add_year("2025")   # crashes only when run

# After: with type hints
def add_year(year: int) -> int:
    return year + 1

add_year("2025")   # mypy error: expected int, got str

Strict mypy (the --strict setting) turns on the toughest checks at once — for example, requiring every function to have type hints. The mypy docs say that with strict mode “you will basically never get a type related error at runtime without a corresponding mypy error.” It is stricter, but it makes a big codebase far safer to change.

9. What is full-text search, FTS5 and BM25?

Plain English: Full-text search finds documents that contain the words you type, quickly, even across millions of records.

The trick is an index. An index is a lookup table that maps each word to the list of records containing it — so the computer never has to scan every record.

FTS5 is SQLite’s built-in full-text search feature. You create a special virtual table and query it with the MATCH keyword:

SELECT title
FROM papers_fts
WHERE papers_fts MATCH 'computer virus'
ORDER BY rank;

BM25 (“Best Matching 25”) is the scoring formula that decides which matches are best. It grew out of the probabilistic retrieval work of Stephen Robertson, Karen Spärck Jones and colleagues in the 1970s–80s, first used in the Okapi system. Intuitively it weighs three things: how often your words appear in a record (but with diminishing returns, so the tenth “virus” adds less than the first), how rare a word is across all records (rare words like “Grassi” are far more telling than common words like “the”), and record length (so short, focused records are not unfairly beaten by long ones). In SQLite’s FTS5 a lower BM25 score means a better match, so you sort ascending.

10. What is fuzzy matching?

Plain English: Fuzzy matching finds text that is close to what you typed, even if it is not spelled exactly the same.

The core idea is edit distance (also called Levenshtein distance, after Vladimir Levenshtein who defined it in 1965): the smallest number of single-character changes — insert, delete, or substitute one letter — needed to turn one word into another.

11. What are embeddings and hybrid ranking?

Plain English: An embedding turns a piece of text into a list of numbers (a vector) in such a way that texts with similar meaning get similar numbers. Closeness of the numbers means closeness of meaning.

We measure how close two vectors are with a similarity score (commonly cosine similarity). This finds papers that mean the same thing even when they share no exact words.

Hybrid ranking combines several signals — exact identifier matches, BM25 keyword scores, fuzzy scores, and embedding similarity — into one final ranking. Each method is strong where the others are weak: BM25 excels at exact and rare-word matches, embeddings handle paraphrases and meaning, and fuzzy matching handles typos. Together they are more reliable than any one alone.

12. What is evaluation and benchmarking?

Plain English: Evaluation means testing your finished system against a known list of tricky inputs and measuring how often it gets the right answer.

You build a benchmark: a collection of messy citations where you already know the correct paper. You run the system over all of them and score its accuracy.

One subtle but important idea: abstention. When the system is not confident, it should say “no confident match” rather than guess. A confident wrong answer is more harmful than an honest “I don’t know”, because a wrong citation can mislead a reader for years.