Skip to content

Project ChargedUp – Roadmap for BatteryProc Major

Update

Project ChargedUp is the code name for the next-generation update of BatteryProc, our internal battery test data processing tool. BatteryProc (v3.x) is a Python CLI program that automates extraction of key battery metrics from Neware .xlsx files, runs various calculations, and then generates plots for analysis[1]. As the organization grows, ChargedUp will evolve this tool into a faster, more flexible, and more maintainable platform. Below is a strategic-level roadmap of improvement areas and new features under consideration in this planning phase. (Data ingestion and processing efficiency are top priorities, whereas items like database integration are exploratory for the longer term.)

Robust Data Ingestion and Format Support

  • Flexible Excel Header Handling: Improve how the parser handles .xlsx files with inconsistent sheet or column headers. Currently the program expects specific sheet names (record, cycle, unit) and exact column headings (e.g. Cycle Index , Voltage(V) , Current(mA) ) – missing or variant names can cause processing warnings or failures[2]. The update will introduce a more tolerant import layer that maps alternate header names to the expected ones (using configurable synonyms and regex matching). This includes enhancing the existing column standardization utilities to auto-detect common variations (e.g. treating "Capacity(mAh)" and "Cap.(mAh)" as equivalent for raw capacity). By making the input parsing more robust, ChargedUp will gracefully handle files with minor format differences without manual pre-cleaning.

o User note: We’ve seen several bugs related to mismatches in file naming in
the past. I am definitely aware of issues that were related to time and current
variables. Those were fixed but are good examples of some of the issues that
have been seen in header problems

o Care is required to check that all portions of the program use the correct
variable names and there aren’t any legacy issues that get carried
through.
- Support for Multiple File Formats and Instruments: Extend BatteryProc beyond Neware Excel files to accept additional data formats. Right now, the CLI strictly filters for .xlsx inputs and will error out if other file types are provided[3], even though internal code has some support for CSV/TXT/MPT. We plan to implement a modular import engine that can handle CSV, text, and other vendor-specific formats (e.g. Arbin or Bio-Logic data files). This might involve creating parser classes or functions for each format and using an extension-based dispatch (so a .csv or .mpt file is recognized and processed appropriately). The system could

auto-detect instrument type from file content and apply the correct parsing rules.
By broadening format support, ChargedUp will let us process data from other
battery cyclers beyond Neware, increasing the tool’s utility. While we might not
have all file types today, we want the program to be easily expanded to handle new
file formats in the future.
- Faster Data Parsing & Performance: Address the data ingestion speed, which is currently a known bottleneck. Large Excel files can be slow to load with pandas/openpyxl. We will explore optimizations such as using faster backends (e.g. the pyxlsb engine or CSV conversion), reading only necessary columns, and leveraging pandas chunking for very large datasets. Another idea is to enable parallel processing of files – for example, if a directory of many files is given, ChargedUp could distribute files across multiple CPU cores to speed up batch processing. Improving I/O efficiency and utilizing concurrency where possible will significantly reduce total run times for large data sets. - Local Database Integration (Exploratory): In the longer term, consider adding an option to store processed data in a lightweight local database (e.g. SQLite). This would allow ChargedUp to cache test results and metadata for quick lookup and aggregation. For example, after parsing an .xlsx, key metrics could be saved to a database table, enabling users to query historical results or combine data without reprocessing raw files. This idea is still preliminary – initial phases will focus on core parsing improvements – but a data storage layer could become useful as our library of test results grows. We will sketch out how a local DB or simple data repository might integrate with the tool (likely as an optional step after file processing), ensuring it doesn’t over-complicate the basic file-in/file-out workflow.

Formula-Driven Calculations for Consistency

  • Decision (2025-10-21): Canonical cycle identification now lives in batteryproc.formulas.detect_cycle_boundaries with validation/phase helpers, backed by CLI overrides (--cycle-detection, --current-threshold, --validate-cycles) and the merged config at batteryproc/config/cycle_detection.yml. This closes Issue #277 and records the architecture decision under ChargedUp EPIC #184 so downstream metrics stay aligned on tiered detection outputs.

  • Decision (2025-11-03): dQ/dV and parser-driven Cycle Index synthesis now route through the shared configuration-aware detection pipeline. Auto current-threshold tuning metadata is propagated on outputs, and dQ/dV plots preserve this metadata for downstream traceability. (Issue #299 → Epic #296)

  • Decision (2025-01-07): CLI metric reconciliation will continue to prefer file-reported per-cycle summaries when present, falling back to recomputed values when files lack a column or breach tolerance. Default tolerances are set to 5% relative with absolute guardrails of 0.05 mAh (or mAh/g) for capacity metrics, 0.10 mWh (or mWh/g) for energy metrics, and 0.5 percentage points for efficiency-style percentages. These thresholds match current processing outputs while surfacing materially divergent records. The CLI ships --recompute-all, --prefer-file-metrics, --metric-tolerance, and --metric-tolerance-abs so operators can override the defaults on a per-run basis without editing code.
  • Progress (2025-10-29): capacity_summary, energy_summary, and dqdv now execute through the formulas engine via _prepare_normalized_inputs. The adapter preserves the legacy schema, caches cycle-detection metadata, and falls back to the historical pandas routines when normalization trims rows or mass metadata is missing. Averaging helpers and the bad-cell check were updated to consume the canonical column names so follow-on analytics stay consistent with Issue #231's formulas delegation plan.
  • Derived Metrics from Base Variables: Increase the use of calculated formulas for key performance metrics instead of relying solely on values directly from input files. The goal is to improve consistency and trustworthiness of the results. For instance, rather than blindly trusting every capacity or energy value provided in the test file, ChargedUp could recompute certain metrics from primary data (voltage, current, time) to double-check or fill in missing values. Examples include calculating charge/discharge capacity by integrating current over time (ensuring consistency with reported capacity) and computing energy by integrating \(V \times I\) over time if energy columns are absent. The current BatteryProc already demonstrates this philosophy in places – for example, it can derive a unified cycle count from half- cycle data if needed[4] – and we plan to extend it. By basing outputs on a few fundamental inputs (current, time, voltage, etc.), we reduce errors from quirky file exports and ensure uniform calculations across different data sources.

o Mandatory Action: As part of the improvements to metric calculations, our
team needs to research the open source batterypy package to learn if there
is anything that we could learn from or leverage in this development project.
- Automated Data Corrections: Leverage these formulas to automatically correct or flag data issues. If a file’s summary sheet has non-sequential cycle indices or if a specific capacity column appears mis-scaled, ChargedUp will detect the anomaly and use raw data to regenerate the expected values. This might involve normalizing cycle numbering (which BatteryProc already attempts[5]) and ensuring that Coulombic efficiency is computed as \((\text{discharge capacity} / \text{charge capacity})\times100\%\) consistently. These calculated metrics will then feed into plots and reports, giving users confidence that results are internally consistent even if input files have minor issues. Overall, emphasizing formula-driven processing will make the analysis more reliable and reproducible across varied datasets. - Averaging Tool Improvements: Improve maintainability of the averaging tool through a thorough refactoring of the code. Update behavior/functionality of the tool through more flexibly handling of file grouping. Currently the average flag often skips files that should be included in the average group because of tiny file naming errors.

Enhanced User Interface and Experience

  • Improved CLI Usability: Even as we add a GUI (see below), the command-line interface will remain important. We plan to polish the CLI output and user experience. This includes providing clearer status messages and progress indicators when processing multiple files (so users aren’t left guessing during long runs), and printing more informative warnings/errors. For example, if a file is skipped for having insufficient data, the message will explicitly state the reason (cycle count too low, missing sheet, etc.). The current version has already been tidied in places (e.g. cleaned up output messaging), but further tweaks will ensure every log or warning is actionable and easy to understand [6]. Additionally, we’ll review and update command-line options for consistency and intuitiveness, possibly adding shortcuts for common flag combinations. These refinements will make the tool feel more responsive and user-friendly for our engineers running analyses via terminal.
  • Graphical User Interface (GUI): To accommodate users less comfortable with the command line, we will introduce a simple GUI front-end for BatteryProc. This GUI will allow users to select input files/folders and output locations through dialog boxes , toggle processing options (like enabling energy plots or averaging), and run the analysis with the click of a button. Under the hood, implementing the GUI will require refactoring the current code so that the core “process X -> output Y” logic is callable as a function, decoupled from direct argparse CLI handling[7]. We envision using a lightweight toolkit (perhaps Tkinter for minimal dependencies, or a

framework like PySimpleGUI for quicker development) to create a small window
with the necessary controls. When the user hits “Run”, the GUI will call our
processing function (possibly in a background thread to keep the interface
responsive) and then display a summary or success message when done. This
feature will greatly lower the barrier for new team members to use ChargedUp ,
making battery data analysis accessible with a few clicks.
- Visualization and Plotting Enhancements: Continue improving the quality and adaptability of generated plots. BatteryProc already produces publication-ready charts with a consistent style, but we have identified a few areas to enhance: - Adaptive Plot Layout: Implement logic to automatically switch certain plot dimensions or formats when dealing with extreme cases (for example, if a cell has hundreds of cycles , use a wider rectangular plot to accommodate the long x-axis). A feature is on the table to toggle between square and rectangular plot sizes once cycle count exceeds a threshold (e.g. >100 cycles)[6]. This will ensure plots remain clear and legible for both small and very large datasets. - Styling Updates: Apply another round of style refinements to keep the visuals polished. This might include adjusting color schemes or markers for better clarity, refining axis labels, and ensuring consistency across all plot types (capacity vs. cycle, energy vs. cycle, dQ/dV, etc.). We will leverage our style guide and possibly newer Matplotlib capabilities to maintain a modern, cohesive look. - Output Options: Explore if additional output formats or report summaries would be helpful. For instance, ChargedUp could generate a brief PDF report compiling key plots and metrics for each run, or an interactive HTML report. These ideas are optional nice-to-haves; the primary focus will be on perfecting the core PNG/SVG plots and making sure they convey the needed information at a glance.

Quality Assurance: Testing and Code Health

  • Expanded Test Coverage: We plan to significantly increase the automated test coverage of the project to >90%. The current test suite has improved recently but still has gaps, especially in newer features and edge-case scenarios. As we refactor and add functionality, we will concurrently write comprehensive tests for all critical logic – parsing, metrics calculations, plotting, and the CLI/GUI flows. This includes creating test cases for files with missing columns or odd data to ensure our new robust handling works as expected. By aiming for high coverage (and specifically targeting previously untested paths[8]), we gain confidence that ChargedUp changes won’t break existing capabilities. Continuous integration will be set up (or updated) to run the full test suite on each commit. Ultimately, a stronger test suite will guard against regressions and serve as living documentation of expected behaviors.
  • Linting and Code Refinement: We will perform thorough linting and refactoring passes to improve code quality and maintainability. This entails fixing any stylistic

issues or potential bugs flagged by linters (like Pylint or Flake8) and adhering to best
practices in the codebase. For example, some internal functions are currently quite
complex (the _validate_sheets_and_parse() function in cli.py is noted for its
high complexity) – these may be broken into smaller, focused functions for clarity.
We’ll also eliminate code smells such as duplicated logic and overly long modules.
Part of this effort is to restructure the code into a cleaner architecture : separating
concerns so that file parsing, data processing, and plotting are modular units. One
concrete step will be refactoring the monolithic CLI script so that core processing
logic can be invoked programmatically (which also supports the GUI effort). By the
end of linting/refactoring, we expect the codebase to conform to modern Python
standards (PEP8 compliant, with type hints and docstrings where appropriate),
making it easier for both our team and AI coding assistants to navigate and
contribute to the project.
- Architectural Improvements: Beyond surface-level code cleanup, we’ll consider deeper architectural enhancements to boost reliability and maintainability. This may include introducing a clearer object-oriented design – for instance, defining classes like BatteryTestFile or parser objects for each format, which encapsulate all related functions. A plugin-style architecture could allow new instrument support to be added by simply registering a new parser class, without altering the core logic. We’ll ensure that state and data flow through the program are well- defined (perhaps using data classes to package results like capacity statistics, instead of loosely passing DataFrames around). Additionally, performance-oriented architecture changes will be evaluated – for example, if we implement multi- threading for file processing, we might use Python’s concurrent.futures or multiprocessing with proper safeguards (avoiding GIL contention for CPU-bound parts, etc.). The overarching aim is to align ChargedUp with software best practices so that it’s robust, scalable, and easy to extend in the future. As a side benefit, a clean architecture will make the code more approachable for automated tools and new developers, accelerating future development cycles. - Typed Tier Roadmap: Consolidate mypy enforcement into a typed-core override (CLI, metrics, parsers, plots, utils, batteryproc_types) with disallow_untyped_defs = true, keep legacy modules such as batteryproc.colours under an ignore_errors override, and document the promotion pathway for contributors (Issues #151, #147). Maintain the temporary global ignore_missing_imports while we build the stub/caching pipeline from Issue 147 so we can migrate to per-module ignores once official stubs for openpyxl, seaborn, tabulate, and PyYAML are available.

Documentation and Repository Cleanup

  • Up-to-Date Documentation: As we implement these changes, we will update and expand documentation to guide users and contributors. The README and User Guide will be revised to reflect new features (e.g. supported file types, GUI usage instructions) and to clarify any altered behaviors. In particular, documentation will explicitly list the required or expected columns and any accepted variants for different data sources, so users know how to prepare their files[9]. We’ll include troubleshooting tips for common issues (for example, what to do if a file is skipped due to missing data). If we add significant functionality (like database integration or advanced formulas), those will be explained in the docs with examples.
  • Internal Docs and Guides: Alongside user-facing docs, we may create or update developer documentation (such as a Contributing Guide or a design overview) to

help the team understand the new code structure. For instance, if we introduce a
plugin system for parsers or a new config file format, a short guide on how that
works will be valuable. This ensures the whole team can get up to speed on
ChargedUp improvements and follow consistent practices (perhaps captured in an
updated style guide or coding standard document).
- Repo Cleanup: We’ll perform general housekeeping on the repository. This includes removing any obsolete or deprecated scripts, organizing directories (maybe grouping sample data or results in a designated folder), and ensuring naming consistency. All configuration files (for linting, testing, CI, etc.) will be checked in and clearly configured. We will also review the issue tracker and TODO lists, closing or updating items that have been addressed. The end result will be a clean, well-organized repository that new team members can navigate easily. This cleanup, coupled with improved documentation, will make ChargedUp more maintainable and inviting for collaboration.

Packaging and Deployment as a Library

  • Pip Installable Package: A major long-term goal is to transform BatteryProc/ ChargedUp into a proper Python package that can be easily installed and reused. We will set up the necessary packaging infrastructure (likely a setup.py or pyproject.toml configuration with appropriate metadata). This involves defining entry points for the CLI (so that running pip install batteryproc allows users to invoke the batteryproc command from anywhere) and including any console script for the GUI if we add one[10]. We’ll ensure all dependencies are listed and that the package can be installed in a clean environment without issues. During development, we can use editable installs (pip install -e .) to test this packaging.
  • Publishing and Versioning: Once the package setup is solid, we aim to publish the tool to a PyPI index (this could be a private PyPI for internal use, or potentially public if we decide to open-source parts of it). Deployment automation will be configured, possibly via GitHub Actions, to build and upload a new version whenever we tag a release. We will increment the version number to mark this major update (likely going to 4.0.0 given the scope of changes). By packaging ChargedUp , we make distribution within the company much easier – anyone can install or upgrade the tool with a simple pip command, and it lays the groundwork for integration into other systems or pipelines. Moreover, being a package means parts of the code can be imported as a library in notebooks or other Python scripts, enabling more flexible use of the underlying functionality.
  • Continuous Deployment Readiness: In addition to packaging, we’ll consider how ChargedUp could be deployed in different contexts. For example, ensuring it runs cross-platform (Windows, macOS, Linux) as before, and possibly containerizing the application for use in cloud or CI environments. While not immediate, we might
create a Dockerfile or similar so that a fully configured environment (with all
dependencies and perhaps even a headless mode for plotting) can be launched
consistently. Laying this groundwork will make it easier to incorporate ChargedUp
into automated workflows down the line (for instance, automatically processing
test data as it’s produced in the lab).

Project ChargedUp – Vision and Purpose

(Code Name: ChargedUp) This major update initiative was born from our need to scale and modernize BatteryProc for the future. ChargedUp will serve as a key internal tool for our organization, empowering our team to efficiently process and analyze an ever-growing volume of battery test data. By improving reliability, speed, and usability, ChargedUp ensures that as our company expands, our battery data analysis remains accurate and accessible – fueling better decisions and innovation in our battery development efforts.

Sources:

  1. BatteryProc README – tool overview and current features[1][2]
  2. BatteryProc CLI code – input handling and format assumptions[3]
  3. BatteryProc utils and metrics – examples of derived calculations and data handling[4]
  4. Internal TODO and issues – planned GUI and enhancements (test coverage, plotting adjustments, etc.)[7][6][8]
  5. BatteryProc documentation – need for clarifying input requirements and variants[9]

[1] [5] README.md

https://github.com/nickbashian/batteryproc/blob/45eb4dae0380d0c41b007400ecc85ce 136e9f6e/README.md

[2] [9] Warnings Related to Missing Required Columns During Data File Processing

https://github.com/nickbashian/batteryproc/issues/

[3] cli.py

https://github.com/nickbashian/batteryproc/blob/45eb4dae0380d0c41b007400ecc85ce 136e9f6e/batteryproc/cli.py

[4] utils.py

https://github.com/nickbashian/batteryproc/blob/45eb4dae0380d0c41b007400ecc85ce 136e9f6e/batteryproc/utils.py

[6] [7] [8] [10] TODO.md

https://github.com/nickbashian/batteryproc/blob/45eb4dae0380d0c41b007400ecc85ce 136e9f6e/docs/TODO.md