iBet uBet web content aggregator. Adding the entire web to your favor.
iBet uBet web content aggregator. Adding the entire web to your favor.



Link to original content: http://github.com/Toloka/crowd-kit/issues/86
[FEATURE] Unifying classification aggregators interfaces · Issue #86 · Toloka/crowd-kit · GitHub
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEATURE] Unifying classification aggregators interfaces #86

Open
alexdremov opened this issue Oct 5, 2023 · 3 comments
Open

[FEATURE] Unifying classification aggregators interfaces #86

alexdremov opened this issue Oct 5, 2023 · 3 comments
Labels
enhancement New feature or request

Comments

@alexdremov
Copy link
Contributor

alexdremov commented Oct 5, 2023 •
edited
Loading

Problem description

I wanted to test quality metrics of several different algorithms from crowdkit.aggregation.classification and found myself writing such kind of function:

def get_scores(model, data, fit=True):
    if fit:
        model.fit(data)
    probas = getattr(model, "probas_", None)
    if probas is not None:
        return probas
    predictor = getattr(model, "predict_score", None)
    if predictor is None:
        predictor = model.predict_proba
    return predictor(data)

That's because different models have different methods for retrieving scores. For example, MMSR has predict_score while almost all others have predict_proba. Some have field probas_ , while others don't.

This seems strange and inconsistent.

Feature description

Unify naming of predict_score functions and presence of probas_ field
@alexdremov alexdremov added the enhancement New feature or request label Oct 5, 2023
@alexdremov alexdremov changed the title [FEATURE] Unifying classification aggregation interfaces [FEATURE] Unifying classification aggregators interfaces Oct 5, 2023
@pilot7747
Copy link
Contributor

Thanks for the issue! Will simply adding predict_proba for M-MSR solve this? I assume we can just pass the scores through the softmax but I don't remember what these scores mathematically mean, need to read into the paper.

@alexdremov
Copy link
Contributor Author

Thanks for the issue! Will simply adding predict_proba for M-MSR solve this? I assume we can just pass the scores through the softmax but I don't remember what these scores mathematically mean, need to read into the paper.

Yes, if all agreggators will have predict_proba, the issue should be considered solved. Also, presence of probas_ field should be made consistent, as it was confusing for me why some models have it, and others don't

Also, KOS aggregator does not have any probas at all. Is it algorithm's limitation or this can be added?

@dustalov
Copy link
Collaborator

KOS does not operate with probabilities explicitly, but we can always throw NotImplementedError if we find no reasonable workaround.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@dustalov @pilot7747 @alexdremov