Metadata-Version: 2.1
Name: explainaboard
Version: 0.8.7
Summary: Explainable Leaderboards for Natural Language Processing
Home-page: https://github.com/ExpressAI/ExplainaBoard
Author: Pengfei Liu
License: MIT License
Platform: UNKNOWN
Classifier: Intended Audience :: Developers
Classifier: Topic :: Text Processing
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3 :: Only
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: datalabs (>=0.3.7)
Requires-Dist: datasets (>=2.0.0)
Requires-Dist: eaas (>=0.3.8)
Requires-Dist: lexicalrichness
Requires-Dist: matplotlib
Requires-Dist: nltk (>=3.2)
Requires-Dist: numpy
Requires-Dist: pandas
Requires-Dist: pyarrow
Requires-Dist: sacrebleu
Requires-Dist: scikit-learn
Requires-Dist: scipy
Requires-Dist: seqeval
Requires-Dist: spacy
Requires-Dist: tqdm
Requires-Dist: wheel
Provides-Extra: dev
Requires-Dist: pre-commit ; extra == 'dev'

# ExplainaBoard: An Explainable Leaderboard for NLP
[**Introduction**](#introduction) | 
[**Web Tool**](https://github.com/neulab/ExplainaBoard#web-based-toolkit-quick-learning) |
[**API Tool**](#api-based-toolkit-quick-installation) |
[**Download**](#download-system-outputs) |
[**Paper**](https://arxiv.org/pdf/2104.06387.pdf) |
[**Video**](https://www.youtube.com/watch?v=3X6NgpbN_GU) |
[**Bib**](http://explainaboard.nlpedia.ai/explainaboard.bib)

<p align="center">
  <img src="./fig/logo-full-v2.png" width="800" class="center">
  <br />
  <br />
  <a href="https://github.com/neulab/ExplainaBoard/blob/main/LICENSE"><img alt="License" src="https://img.shields.io/github/license/neulab/ExplainaBoard" /></a>
  <a href="https://github.com/neulab/ExplainaBoard/stargazers"><img alt="GitHub stars" src="https://img.shields.io/github/stars/neulab/ExplainaBoard" /></a>
  <a href="https://pypi.org/project//"><img alt="PyPI" src="https://img.shields.io/pypi/v/explainaboard" /></a>
  <a href="https://github.com/psf/black"><img alt="Code Style" src="https://img.shields.io/badge/code%20style-black-black" /></a>
  <a href=".github/workflows/ci.yml"><img alt="Integration Tests", src="https://github.com/neulab/ExplainaBoard/actions/workflows/ci.yml/badge.svg?event=push" /></a>
</p>

## What is ExplainaBoard?

When developing a natural language processing (NLP or AI) system, often one of the hardest things is to understand where your system is working and where it is failing, and deciding what to do next. ExplainaBoard is a tool that *inspects your system outputs*, *identifies what is working and what is not working*, and helps *inspire you with ideas of where to go next*.

It offers a number of different ways with which you can evaluate and understand your data:

1. *Single-system Analysis*: What is a system good or bad at?
2. *Pairwise Analysis*: Where is one system better (worse) than another?
3. *Data Bias Analysis*: What are the characteristics of different evaluated datasets?
4. *Common Errors*: What are common mistakes that top-5 systems made?
5. *Fine-grained Error Analysis*: where do errors occur?
6. *System Combination*: Is there potential complementarity between different systems?

<img src="./fig/intro.png" width="400" class="center">

How do you get started?

1. *Browse Outputs Online*: We provide an [online interface](http://explainaboard.nlpedia.ai/) where you can browse and understand outputs from various state-of-the-art systems.
2. *Upload Your Own Outputs*: You can also [upload your own system outputs](https://docs.google.com/forms/d/e/1FAIpQLSdb_3PPRTXXjkl9MWUeVLc8Igw0eI-EtOrU93i6B61X9FRJKg/viewform) through the online interace, and we will process them so you can view them alongside other systems.
3. *Run Your Own Analyses Offline*: You can follow the installation instructions below to run analyses on your own system data offline. We also plan to [support command-line upload to the web interface](https://github.com/neulab/ExplainaBoard/issues/60) soon!


## Web-based Toolkit


We have deployed ExplainaBoard as a [Web toolkit](http://explainaboard.nlpedia.ai/), which includes 9 NLP tasks and will be more,
40 datasets and 300 systems. Detailed information is as follows.
<img src="./fig/demo.gif" width="800" class="center">

### Installation

**Method 1 - Standard Use:** Simple installation from PyPI (Python 3 only)

```
pip install --upgrade pip  # recommending the newest version of pip.
pip install explainaboard
python -m spacy download en_core_web_sm  # if you plan to use the TextClassificationProcessor
```

**Method 2 - Development:** Install from the source and develop locally (Python 3 only)

```bash
# Clone current repo
git clone https://github.com/neulab/ExplainaBoard.git
cd ExplainaBoard

# Install the required dependencies and dev dependencies
pip install ."[dev]"
pre-commit install
```

- **Testing:** To run tests, you can run `python -m unittest`. 
- **Linting and Code Style:** This project uses flake8 (linter) and black (formatter). They are enforced in the pre-commit hook and in the CI pipeline.
    - run `python -m black .` to format code
    - run `flake8` to lint code
    - You can also configure your IDE to automatically format and lint the files as you are writing code.
    - There are various ways to skip formatting or linting if you find it necessary. The following are some examples (please refer to official documentation for more options)
        - `# noqa` signals flake8 to ignore the current line. (`flake8: noqa` ignores the current file.)
        - lines between `# fmt: on` and `# fmt: off` are skipped by black. 

Then, you can run following examples via bash


### Example for CLI

* text-classification:
```shell
explainaboard --task text-classification --system_outputs ./data/system_outputs/sst2/sst2-lstm.tsv
```
See detailed [description](https://github.com/neulab/ExplainaBoard/blob/main/docs/task_text_classification.md) and 
more [supported tasks](https://github.com/neulab/ExplainaBoard/blob/main/docs/supported_tasks.md).


### Example for Python SDK

```python
from explainaboard import TaskType, get_loader, get_processor

path_data = "./explainaboard/tests/artifacts/test-summ.tsv"
loader = get_loader(TaskType.summarization, data=path_data)
data = list(loader.load())
processor = get_processor(TaskType.summarization)
analysis = processor.process(metadata={}, sys_output=data)
analysis.write_to_directory("./")
```


## Acknowledgement
We thanks all authors who share their system outputs with us: Ikuya Yamada, Stefan Schweter,
Colin Raffel, Yang Liu, Li Dong. We also thank
Vijay Viswanathan, Yiran Chen, Hiroaki Hayashi for useful discussion and feedback about ExplainaBoard.



