Data Science Script Library

1What This Library Is For Core

The Data Science Script Library serves two purposes at once [1]: practical script utility for students doing data science, analytics, and Machine Learning (ML) coursework, and a deliberate demonstration of CodeSentinel maintenance discipline -- the same baseline-diffing, strict-mode quality gating this repository's own tooling practices, applied here at a beginner-friendly scale. Section 10 returns to that second purpose directly.

This guide covers both halves. Sections 1-5 get a beginner from zero to a first successful script run, including choosing between the Core, Full, and TensorFlow dependency profiles. Sections 6-8 add a worked example for every family, a realistic chained workflow, and how to diagnose a dependency conflict. Sections 9-10 are optional advanced material on batch automation and on the repository's own maintenance script.

Prerequisite: basic familiarity with running a Python script from a project directory. If that is not yet comfortable, start with Virtual Environments, Jupyter, and VS Code first.

2Script Families at a Glance Core

The source repository groups its scripts by workflow, not by package internals. All seven families below are adopted here; the last column points ahead to each family's worked example in section 6.

3Choosing a Dependency Profile Core

The scripts above split their dependencies into three install profiles. Pick the smallest one that covers the assignment.

Selection rule [2]: start with Core for notebook and report basics, move to Full only when an explicit feature requires it, and capture pip freeze when reproducibility matters. Section 8 covers what to do when a script's imports do not match the profile that was actually installed.

4Student Setup Core

Work inside an activated project virtual environment before installing anything. If virtual environments, Jupyter, or VS Code setup are still unfamiliar, start with Virtual Environments, Jupyter, and VS Code first -- this guide assumes that part is already working.

python -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -r requirements.txt

The repository also ships a non-interactive setup script that wraps this same choice [3]. Its default behavior is safe: it installs the Core profile unless told otherwise.

python scripts/repo/setup/setup_student_env.py --deps core

Swap --deps core for --deps full or --deps tensorflow-class to match the profile chosen in section 3. Add --interactive for prompt-driven setup instead of the non-interactive default.

5Running Your First Script Core

csv_profile_report.py is a good first script: it only needs the Core profile, takes one input file, and produces a report you can open and read immediately.

Start with any CSV file that has a header row -- a small gradebook export works fine. Run it against an output directory:

python scripts/data/csv_profile_report.py grades.csv --out report_out

The script prints the two files it wrote:

Wrote: report_out/csv_profile.md
Wrote: report_out/csv_profile.html

Open csv_profile.html in a browser, or csv_profile.md in any Markdown viewer. Either report contains row and column counts, per-column missing and unique counts, numeric statistics (minimum, maximum, mean) for columns that parse as numbers, and a small sample of the first rows -- enough to sanity-check a dataset before doing anything else with it.

6A Worked Example per Script Family Intermediate

One representative script per family, with its real command-line arguments and what it actually prints or writes. Each script has more options than shown here; run it with --help for the full list.

Benchmarks -- benchmark_simulator.py

Runs latency micro-benchmarks against a JSON graph file: random-access lookups, neighbor traversal, and shallow breadth-first path finding [4].

python scripts/benchmarks/benchmark_simulator.py --graph campus_graph.json --seed 7 --json

Without --json, it prints three text sections -- Random Access, Neighbor Traversal, and Path Finding -- each with p50/p95/p99 latency in milliseconds. With --json, it emits one JSON payload with the same results instead, useful when a later step needs to parse the numbers rather than read them.

Data -- csv_profile_report.py

Already introduced in section 5. --max-rows, --delimiter, and --sample-rows are worth knowing for a larger or unusually formatted file [5].

python scripts/data/csv_profile_report.py survey.csv --out report_out --delimiter ";" --sample-rows 10

Docs -- convert_to_pdf.py

Converts a Markdown file to a styled PDF, preserving headings, code blocks, tables, lists, and links [6]. It takes its two file paths as plain positional arguments rather than named flags.

python scripts/docs/convert_to_pdf.py report.md report.pdf

On success it prints Successfully converted report.md to report.pdf. Run it with no arguments to see the usage line; a missing input file prints Error: Input file report.md does not exist instead of a Python traceback.

ML -- run_ml_pipeline_demo.py

Runs a complete synthetic pipeline end to end: generate synthetic records, build a feature dataset, train a model, and score the records [7]. It needs no real dataset of your own -- a good first ML script to run.

python scripts/ml/run_ml_pipeline_demo.py --out-dir ml_demo --seed 7

It writes input/telemetry.jsonl (the synthetic records), a dataset/ directory, a model/ directory, and scores/scores.csv under the output directory, then prints Pipeline demo complete under: ml_demo. It exits with code 2 if any stage fails, which section 9 puts to use.

Notebooks -- export_notebook.py

Exports a .ipynb notebook to HTML or PDF [8], printing a step for each stage so a failure is easy to locate.

python scripts/notebooks/export_notebook.py analysis.ipynb --format html --output analysis.html

Console output runs Reading: analysis.ipynb, then Converting to HTML..., then Writing: analysis.html, then Success!. A missing file prints Error: File not found: instead of failing silently.

Plots -- plot_timeseries_from_csv.py

Draws a line chart from CSV data with sensible defaults, for quick plotting during coursework [9]. If --y is omitted, it plots every numeric-looking column against the x-axis automatically.

python scripts/plots/plot_timeseries_from_csv.py weekly_scores.csv --out weekly_scores.png --x week --y average_score --title "Average score by week"

It writes the PNG and prints Wrote: weekly_scores.png.

Repo helpers -- generate_script_inventory.py

Scans a directory for scripts and writes a structured inventory of what it finds [10]. Section 10 uses this same script's output as part of the repository's own maintenance routine.

python scripts/repo/inventory/generate_script_inventory.py --root scripts --out inventory_out --use-git

It writes inventory_out/script_inventory.json (machine-readable: generation timestamp, script count, and one entry per script with path, size, modification date, and, with --use-git, its last-commit timestamp) and inventory_out/script_inventory.md (the same data as a readable report), printing Wrote: for each.

7Combining Families in a Real Workflow Intermediate

A real assignment rarely stops at one script. This walks through profiling a raw dataset, plotting what the profile turned up, and handing in the result as a notebook export -- three families chained the way an actual assignment would use them.

1. Profile the raw data

   Run csv_profile_report.py first, before touching anything else. Its per-column numeric statistics and missing-value counts are what tell you which columns are even worth plotting.

   Step 1 -- profile

   python scripts/data/csv_profile_report.py weekly_scores.csv --out workflow_out/profile

2. Plot the column the profile flagged

   Point plot_timeseries_from_csv.py at the same input file and the column the profile step showed was numeric and complete.

   Step 2 -- plot

   python scripts/plots/plot_timeseries_from_csv.py weekly_scores.csv --out workflow_out/plot.png --x week --y average_score --title "Average score by week"

3. Export the analysis notebook to hand in

   With the profile and plot in hand, write up the findings in a notebook, embed the plot, and export it to HTML for a reader who does not have Jupyter installed.

   Step 3 -- export

   python scripts/notebooks/export_notebook.py analysis.ipynb --format html --output workflow_out/analysis.html

The same pattern generalizes: profile before you plot, and export last, once the analysis it documents is actually finished.

8Diagnosing Dependency Conflicts Intermediate

Most dependency problems in this library trace back to one cause: a script from section 6 needs the Full profile, but only Core is installed.

1. Read the symptom

   An ImportError naming matplotlib, pandas, seaborn, scikit-learn, pyarrow, or nbconvert almost always means a Full-only script is running under Core. It is not a broken script.

2. Check which profile is actually installed

   Run pip freeze and look for the missing package's name. If it is absent, the environment was set up with requirements.txt, not requirements-full.txt.

3. Reinstall the correct profile, not the one missing package

   Follow the section 3 selection rule and reinstall requirements-full.txt in full, rather than pip install-ing the one missing package by hand. A hand-patched environment drifts from what pip freeze would show a classmate, which is exactly what the reproducibility note in section 3 is trying to prevent.

4. TensorFlow profile: a version error, not a package error

   If setup_student_env.py --deps tensorflow-class stops with tensorflow-class profile requires Python 3.13 and Provide a Python 3.13 interpreter with --python, the problem is which Python interpreter was selected, not a missing package. Point --python at an actual Python 3.13 interpreter and rerun -- reinstalling packages will not fix a version mismatch.

5. Capture the working environment once it is fixed

   After resolving a conflict, run pip freeze > requirements-lock.txt (or similar) so the exact working set is on record for next time, instead of rediscovering the same conflict later.

9Advanced: Safe Batch Automation and Structured-Output Pipelines Optional advanced

Several scripts in this library already emit structured output rather than free-form text, which is what makes chaining them into a batch pipeline safe rather than fragile.

benchmark_simulator.py --json (section 6) emits one JSON payload instead of formatted text sections -- a downstream step can parse it directly instead of scraping printed numbers. generate_script_inventory.py (section 6) writes a machine-readable script_inventory.json alongside its human-readable Markdown report, for the same reason.

Before chaining scripts in a batch job, apply the same discipline maintain.py uses on itself (section 10): run against a scratch output directory first, check the exit code before moving to the next stage rather than assuming success, and never point two batch runs at the same output directory at once. run_ml_pipeline_demo.py (section 6) already follows this pattern -- it returns exit code 2 if any pipeline stage fails, which a batch script should check explicitly rather than ignore.

10Advanced: Extending and Validating Repo Helpers Optional advanced

maintain.py, at the repository root, is the single-command maintenance wrapper the README's dual-purpose framing (section 1) refers to -- it is the actual CodeSentinel maintenance discipline this repository practices on itself [11].

A full run works through the same steps every time:

1. Surface inventory

   Counts Python files under scripts/ and test files under tests/ -- the same counting the repo-helper family's own generate_script_inventory.py (section 6) performs.

2. Date integrity check

   Scans the changelog for future-dated headers.

3. Tests

   Runs pytest, unless --quick or --skip-tests is set.

4. Duplicate scan

   Runs scripts/repo/analysis/find_duplicate_functions.py [12], unless in quick or dry-run mode.

5. Baseline comparison

   Loads the previous baseline and compares current script and test counts against it.

6. Report generation

   Writes a JSON and a Markdown quality report under report_tmp/maintenance/.

Exit codes follow the same explicit-status convention the ML and Docs scripts use in section 6: 0 for OK (or for warnings without --strict), 1 for a warning under --strict, 2 for a test or duplicate-scan error, and 3 for a wrapper exception. Reading this exit-code contract before extending maintain.py -- rather than only reading its printed output -- is what lets a change to it be validated the same way the rest of the repository validates a change: check the exit code, not just whether something scrolled past on screen.

References

1. [1] Polymath Global. README.md. Project purpose, dual-purpose framing, and the seven script families. Snapshot verified 2026-08-15.
2. [2] Polymath Global. requirements.txt, requirements-core.txt, and requirements-full.txt. Core and Full dependency profiles. Snapshot verified 2026-08-15.
3. [3] Polymath Global. scripts/repo/setup/setup_student_env.py. Setup automation, --deps profiles, and the Python 3.13 TensorFlow requirement. Snapshot verified 2026-08-15.
4. [4] Polymath Global. scripts/benchmarks/benchmark_simulator.py. Snapshot verified 2026-08-15.
5. [5] Polymath Global. scripts/data/csv_profile_report.py. Snapshot verified 2026-08-15.
6. [6] Polymath Global. scripts/docs/convert_to_pdf.py. Snapshot verified 2026-08-15.
7. [7] Polymath Global. scripts/ml/run_ml_pipeline_demo.py. Snapshot verified 2026-08-15.
8. [8] Polymath Global. scripts/notebooks/export_notebook.py. Snapshot verified 2026-08-15.
9. [9] Polymath Global. scripts/plots/plot_timeseries_from_csv.py. Snapshot verified 2026-08-15.
10. [10] Polymath Global. scripts/repo/inventory/generate_script_inventory.py. Snapshot verified 2026-08-15.
11. [11] Polymath Global. maintain.py. Snapshot verified 2026-08-15.
12. [12] Polymath Global. scripts/repo/analysis/find_duplicate_functions.py. The duplicate scanner maintain.py calls in section 10. Snapshot verified 2026-08-15.