API reference

Re-exported at the top level: from atombench import submit, validate_submission, SubmissionError.

Submitting to the leaderboard

atombench.submit.submit(csv, *, dataset, prop, model_name, author_email, project_url, git_url, team_name=None, description=None, new_benchmark=False, splits_json=None, repo='atomgptlab/jarvis_leaderboard', base=None, contribution_name=None, push=True, out_dir=None, token=None, rebuild=False, keep=False, check_structures=True, extra_metadata=None, echo=<built-in function print>)[source]

Validate, build, and (optionally) push a JARVIS-Leaderboard contribution.

Returns a dict with contribution_dir, created (artifact paths), pr_url (or None), and validation (the ValidationReport). See the module docstring for usage. Raises SubmissionError / GitHubError on failure.

Parameters:

  • csv (str)

  • dataset (str)

  • prop (str)

  • model_name (str)

  • author_email (str)

  • project_url (str)

  • git_url (str)

  • team_name (str | None)

  • description (str | None)

  • new_benchmark (bool)

  • splits_json (str | None)

  • repo (str)

  • base (str | None)

  • contribution_name (str | None)

  • push (bool)

  • out_dir (str | None)

  • token (str | None)

  • rebuild (bool)

  • keep (bool)

  • check_structures (bool)

  • extra_metadata (dict | None)

  • echo (Callable[[str], None])

Return type:

dict

atombench._leaderboard.validate_submission(df, dataset, prop, *, benchmark_ids=None, check_structures=True, max_struct_checks=None)[source]

Validate a benchmark DataFrame against the leaderboard’s real requirements.

Catches the failure modes that otherwise produce a silently-zeroed score or a rejected PR: missing columns, empty/duplicate ids, id sets that don’t match the benchmark test split, and predictions that can’t be parsed.

Parameters:

  • df (pandas.DataFrame)

  • dataset (str)

  • prop (str)

  • benchmark_ids (Set[str] | None)

  • check_structures (bool)

  • max_struct_checks (int | None)

Return type:

ValidationReport

class atombench._leaderboard.ValidationReport(ok, errors=<factory>, warnings=<factory>)[source]

Outcome of validate_submission().

Parameters:
exception atombench._leaderboard.SubmissionError[source]

Raised for any user-facing problem while preparing a submission.

Computing metrics

atombench.cli.compute_metrics(df, bench_name, *, amd_k, symprec, kmin)[source]

Compute all reconstruction metrics for one benchmark DataFrame.

Parameters:

  • df (pandas.DataFrame)

  • bench_name (str)

  • amd_k (int)

  • symprec (float)

  • kmin (int)

Return type:

dict

Verifying benchmark consistency

atombench-verify <path> checks that benchmark CSVs for the same dataset share an identical test-set ID list (and warns if target structures disagree). Building blocks:

atombench.verify.group_by_dataset(pairs)[source]

Group (name, csv_path) pairs by dataset tag (alex / jarvis).

Parameters:

pairs (List[Tuple[str, Path]])

Return type:

Dict[str, List[Tuple[str, Path]]]

atombench.verify.compare_group(dataset_key, items, show_diff)[source]

Return True if ID sets are consistent (or differences are fully accounted for by *.misses.csv files); False on unaccounted mismatches.

Parameters:
Return type:

bool