### Install and Import Libraries
Source: https://github.com/toloka/toloka-kit/blob/main/examples/2.spatial_crowdsourcing/0.simplest_example/spatial_crowdsourcing.ipynb
Installs necessary libraries like toloka-kit, pandas, and ipyplot, then imports them for use in the project. This is a prerequisite for running the example.
```python
%%capture
!pip install toloka-kit==0.1.26
!pip install pandas
!pip install ipyplot
import datetime
import io
import logging
import sys
import getpass
import ipyplot
import pandas
from PIL import Image
import toloka.client as toloka
import toloka.client.project.template_builder as tb
```
--------------------------------
### Install Dependencies
Source: https://github.com/toloka/toloka-kit/blob/main/examples/4.ranking/side_by_side_image_comparision/side_by_side_comparision.ipynb
Installs necessary libraries including toloka-kit, crowd-kit, pandas, and ipyplot. This is a prerequisite for running the examples.
```python
%%capture
!pip install toloka-kit==0.1.26
!pip install crowd-kit==1.0.0
!pip install pandas
!pip install ipyplot
```
--------------------------------
### Install Toloka-Kit and IPython
Source: https://github.com/toloka/toloka-kit/blob/main/examples/1.computer_vision/video_collection/video_collection.ipynb
Installs the toloka-kit library version 0.1.26 and IPython for environment setup. This is a prerequisite for using the Toloka API.
```python
%%capture
!pip install toloka-kit==0.1.26
!pip install ipython
```
--------------------------------
### Install Dependencies
Source: https://github.com/toloka/toloka-kit/blob/main/examples/3.audio_analysis/audio_transcription/audio_transcription.ipynb
Installs necessary libraries including toloka-kit, crowd-kit, pandas, numpy, sentence-transformers, and nltk. This is a prerequisite for running the audio transcription example.
```python
%%capture
!pip install toloka-kit==0.1.26
!pip install crowd-kit==1.0.0
!pip install pandas
!pip install numpy
!pip install sentence-transformers
!pip install nltk
import datetime
import sys
import logging
import getpass
import pandas
import numpy as np
from sentence_transformers import SentenceTransformer
import toloka.client as toloka
import toloka.client.project.template_builder as tb
from crowdkit.aggregation import TextHRRASA
```
--------------------------------
### Install Dependencies
Source: https://github.com/toloka/toloka-kit/blob/main/examples/5.nlp/sentiment_analysis/sentiment_analysis.ipynb
Installs the necessary libraries for Toloka-Kit, crowd-kit, and ipyplot. This is a prerequisite for running the sentiment analysis example.
```python
%%capture
!pip install toloka-kit==0.1.26
!pip install crowd-kit==1.0.0
!pip install ipyplot
```
--------------------------------
### Initialize Image Display
Source: https://github.com/toloka/toloka-kit/blob/main/examples/1.computer_vision/image_collection/image_collection.ipynb
Installs the 'ipyplot' library and imports necessary modules for image manipulation and display. This setup is required before displaying results.
```python
!pip install ipyplot
from PIL import Image, ImageDraw
import ipyplot
results_iter = iter(results_list)
```
--------------------------------
### Install Toloka-Kit and IPython
Source: https://github.com/toloka/toloka-kit/blob/main/examples/3.audio_analysis/audio_collection/audio_collection.ipynb
Installs the necessary libraries for interacting with Toloka and for enhanced notebook functionality. This is a prerequisite for running the subsequent code examples.
```python
%%capture
!pip install toloka-kit==0.1.26
!pip install ipython
import datetime
import logging
import sys
import time
import getpass
import toloka.client as toloka
import toloka.client.project.template_builder as tb
import IPython.display as display
import pandas
```
--------------------------------
### New Metrics Example
Source: https://github.com/toloka/toloka-kit/blob/main/CHANGELOG.md
A new example demonstrating the usage of metrics has been added. This example can help users understand how to implement and utilize metrics within their Toloka projects.
```python
# New metrics example
```
--------------------------------
### Install Dependencies
Source: https://github.com/toloka/toloka-kit/blob/main/examples/benchmarks/image_classification_cinic10.ipynb
Installs the toloka-kit, crowd-kit, and ipyplot libraries required for the project. The '%%capture' magic command suppresses output during installation.
```python
%%capture
!pip install toloka-kit==0.1.26
!pip install crowd-kit==1.0.0
!pip install ipyplot # display images
```
--------------------------------
### Install Prefect
Source: https://github.com/toloka/toloka-kit/blob/main/examples/9.toloka_and_ml_on_prefect/example.ipynb
Installs the Prefect library. Use the appropriate command for your environment manager (pip, conda, or pipenv).
```python
!pip install prefect
# !conda install -c conda-forge prefect
# !pipenv install --pre prefect
```
--------------------------------
### Install and Import Libraries
Source: https://github.com/toloka/toloka-kit/blob/main/examples/1.computer_vision/text_recognition/text_recognition.ipynb
Installs required libraries including toloka-kit, crowd-kit, and ipyplot, then imports essential modules for data handling, logging, and Toloka client interaction.
```python
%%capture
!pip install toloka-kit==0.1.26
!pip install crowd-kit==1.0.0
!pip install ipyplot
import datetime
import os
import sys
import time
import logging
import getpass
import ipyplot
import pandas
import numpy as np
import toloka.client as toloka
import toloka.client.project.template_builder as tb
from crowdkit.aggregation import ROVER
```
--------------------------------
### Create Prefect Project and Start Local Agent
Source: https://github.com/toloka/toloka-kit/blob/main/examples/9.toloka_and_ml_on_prefect/example.ipynb
Creates a new Prefect project and starts a local agent to run Prefect flows on your machine. The agent monitors and executes flow runs.
```bash
PROJECT_NAME = input()
# PROJECT_NAME = 'Toloka test project 1'
!prefect create project $PROJECT_NAME
!prefect agent local start
```
--------------------------------
### Installing Toloka-Kit with S3 Extra
Source: https://github.com/toloka/toloka-kit/blob/main/CHANGELOG.md
Install the 's3' extra to use the `S3Storage` class. This is required for storing and retrieving data using Amazon S3.
```bash
pip install toloka-kit[s3]
```
--------------------------------
### Install and Import Libraries
Source: https://github.com/toloka/toloka-kit/blob/main/examples/3.audio_analysis/audio_transcription/audio_transcription.ipynb
Installs the python-Levenshtein library and imports necessary modules for Levenshtein distance calculation and Toloka streaming.
```python
!pip install python-Levenshtein
import Levenshtein
from toloka.streaming import Pipeline, AssignmentsObserver
```
--------------------------------
### Install Toloka-Kit with Specific Dependencies
Source: https://github.com/toloka/toloka-kit/blob/main/README.md
Install the toloka-kit package with specific optional dependencies. Remove any unnecessary requirements from the list as needed.
```bash
pip install toloka-kit[pandas,autoquality,s3,zookeeper,jupyter-metrics] # remove unnecessary requirements from the list
```
--------------------------------
### Install and Import Libraries
Source: https://github.com/toloka/toloka-kit/blob/main/examples/7.survey/simplest_survey/simplest_survey.ipynb
Installs necessary libraries like toloka-kit, pandas, and plotly. Imports modules required for the survey creation process.
```python
%%capture
!pip install toloka-kit==0.1.26
!pip install pandas
!pip install plotly
import datetime
import sys
import time
import logging
import plotly.express as px
import pandas
import toloka.client as toloka
import toloka.client.project.template_builder as tb
```
--------------------------------
### Install and Import Libraries
Source: https://github.com/toloka/toloka-kit/blob/main/examples/3.audio_analysis/audio_classification/audio_classification.ipynb
Installs necessary libraries including toloka-kit, crowd-kit, and pandas. Imports essential modules for Toloka client, template builder, and data handling.
```python
%%capture
!pip install toloka-kit==0.1.26
!pip install crowd-kit==1.0.0
!pip install pandas
import datetime
import sys
import time
import logging
import getpass
import pandas
import numpy as np
import toloka.client as toloka
import toloka.client.project.template_builder as tb
from crowdkit.aggregation import DawidSkene
```
--------------------------------
### Install and Import Libraries
Source: https://github.com/toloka/toloka-kit/blob/main/examples/1.computer_vision/faces_detection/faces_detection.ipynb
Installs necessary libraries for interacting with the Toloka API, plotting images, and performing data manipulation. Imports essential modules for the project.
```python
!pip install toloka-kit==1.0.0
!pip install ipyplot
!pip install crowd-kit==1.1.0
import os
import datetime
import time
import logging
import sys
import pandas as pd
import ipyplot
from typing import List
from toloka.streaming.event import AssignmentEvent
import toloka.client as toloka
import toloka.client.project.template_builder as tb
from crowdkit.aggregation import MajorityVote
logging.basicConfig(
format='[%(levelname)s] %(name)s: %(message)s',
level=logging.INFO,
stream=sys.stdout,
)
```
--------------------------------
### Install Dependencies and Import Libraries
Source: https://github.com/toloka/toloka-kit/blob/main/examples/1.computer_vision/faces_detection/faces_detection.ipynb
Installs necessary libraries like Pillow for image manipulation and requests for HTTP calls. Imports Image and ImageDraw from Pillow, and requests.
```python
!pip install pillow # To deal with images
!pip install requests # To make HTTP requests
from PIL import Image, ImageDraw
import requests
```
--------------------------------
### Installing Toloka-Kit with Jupyter Metrics Extra
Source: https://github.com/toloka/toloka-kit/blob/main/CHANGELOG.md
Install the 'jupyter-metrics' extra to use the `toloka.metrics.jupyter_dashboard` module. This is necessary for Jupyter-based dashboard functionalities.
```bash
pip install toloka-kit[jupyter-metrics]
```
--------------------------------
### Install Toloka and Crowd-Kit Libraries
Source: https://github.com/toloka/toloka-kit/blob/main/examples/6.streaming_pipelines/streaming_pipelines.ipynb
Installs the necessary Toloka and Crowd-Kit libraries. Ensure you have toloka-kit version 0.1.26 or higher.
```python
%%capture
!pip install toloka-kit==0.1.26
!pip install crowd-kit==1.0.0
```
--------------------------------
### Prepare Environment and Import Libraries
Source: https://github.com/toloka/toloka-kit/blob/main/examples/1.computer_vision/object_detection/object_detection.ipynb
Installs necessary libraries for Toloka API interaction, image plotting, and crowd-kit aggregation. Imports essential modules for data manipulation, API calls, and event handling.
```python
%%capture
!pip install toloka-kit==0.1.26 # To interact with Toloka API
!pip install ipyplot # To plot images inside Jupyter Notebooks cells
!pip install crowd-kit==1.0.0
import os
import datetime
import time
import logging
import sys
import getpass
import pandas as pd # To perform data manipulation
import ipyplot
from typing import List
from toloka.streaming.event import AssignmentEvent
import toloka.client as toloka
import toloka.client.project.template_builder as tb
from crowdkit.aggregation import MajorityVote
logging.basicConfig(
format='[%(levelname)s] %(name)s: %(message)s',
level=logging.INFO,
stream=sys.stdout,
)
```
--------------------------------
### Install and Import Crowd-Kit
Source: https://github.com/toloka/toloka-kit/blob/main/examples/0.getting_started/0.learn_the_basics/learn_the_basics.ipynb
Installs the Crowd-Kit library, version 1.1.0, and imports the MajorityVote class for result aggregation.
```python
!pip install crowd-kit==1.1.0
from crowdkit.aggregation import MajorityVote
```
--------------------------------
### Install Toloka-Kit and Dependencies
Source: https://github.com/toloka/toloka-kit/blob/main/examples/0.getting_started/0.learn_the_basics/learn_the_basics.ipynb
Installs the Toloka-Kit library, pandas, and ipyplot. This is a prerequisite for using the Toloka API in Python.
```python
%%capture
!pip install toloka-kit==1.0.2
!pip install pandas
!pip install ipyplot
```
--------------------------------
### Install Toloka-Kit with All Dependencies
Source: https://github.com/toloka/toloka-kit/blob/main/README.md
Install the toloka-kit package with all additional dependencies using pip. This is useful for environments requiring the full suite of features.
```bash
pip install toloka-kit[all]
```
--------------------------------
### Install Libraries and Import Modules
Source: https://github.com/toloka/toloka-kit/blob/main/examples/1.computer_vision/image_classification/image_classification.ipynb
Installs required libraries including toloka-kit, crowd-kit, pandas, and ipyplot. Imports essential modules for Toloka client, template building, data handling, and logging.
```python
%%capture
!pip install toloka-kit==0.1.26
!pip install crowd-kit==1.0.0
!pip install pandas
!pip install ipyplot
import datetime
import os
import sys
import time
import logging
import getpass
import ipyplot
import pandas
import numpy as np
import toloka.client as toloka
import toloka.client.project.template_builder as tb
from crowdkit.aggregation import DawidSkene
```
--------------------------------
### Start Toloka Pool
Source: https://github.com/toloka/toloka-kit/blob/main/examples/1.computer_vision/text_recognition/text_recognition.ipynb
Opens a training or main pool for task completion. Ensure all configurations are correct before starting.
```python
training = toloka_client.open_training(training.id)
print(f'training - {training.status}')
pool = toloka_client.open_pool(pool.id)
print(f'main pool - {pool.status}')
```
--------------------------------
### Install Toloka Kit and Import Libraries
Source: https://github.com/toloka/toloka-kit/blob/main/examples/1.computer_vision/image_collection/image_collection.ipynb
Installs the toloka-kit library and imports essential modules for interacting with the Toloka API and building task interfaces. This is a prerequisite for most Toloka-related tasks.
```python
%%capture
!pip install toloka-kit==0.1.26
import datetime
import logging
import sys
import time
import getpass
import toloka.client as toloka
import toloka.client.project.template_builder as tb
```
--------------------------------
### Setup AutoQuality Pools
Source: https://github.com/toloka/toloka-kit/blob/main/examples/autoquality/autoquality_usage.ipynb
Call `setup_pools` to create multiple pools with different quality control settings. This method automates the creation of autoquality pools based on the configuration.
```python
aq.setup_pools()
```
--------------------------------
### Create and Open Pool
Source: https://github.com/toloka/toloka-kit/blob/main/examples/1.computer_vision/image_collection/image_collection.ipynb
Creates the task pool using the configured settings and then opens it, making it available for performers to start accepting tasks.
```python
new_pool = toloka_client.create_pool(new_pool)
new_pool = toloka_client.open_pool(new_pool.id)
pool_id = new_pool.id
```
--------------------------------
### Installing Toloka-Kit with ZooKeeper Extra
Source: https://github.com/toloka/toloka-kit/blob/main/CHANGELOG.md
Install the 'zookeeper' extra to use the `ZooKeeperLocker` class. This dependency is required for distributed locking with ZooKeeper.
```bash
pip install toloka-kit[zookeeper]
```
--------------------------------
### Set Project Instructions
Source: https://github.com/toloka/toloka-kit/blob/main/examples/1.computer_vision/image_classification/image_classification.ipynb
Defines the instructions for the task performers in HTML format, guiding them on how to classify images.
```python
project.public_instructions = """Decide what category the image belongs to.
Select "Cat" if the picture contains one or more cats.
Select "Dog" if the picture contains one or more dogs.
Select "Other" if:
- the picture contains both cats and dogs
- the picture is a picture of animals other than cats and dogs
- it is not clear whether the picture is of a cat or a dog
"""
```
--------------------------------
### Download and Preview Dataset
Source: https://github.com/toloka/toloka-kit/blob/main/examples/1.computer_vision/image_classification/image_classification.ipynb
Downloads a toy dataset for image classification and displays a preview of the images and their labels. Ensure you have pandas, ipyplot, and logging libraries installed.
```python
!curl https://tlk.s3.yandex.net/dataset/cats_vs_dogs/toy_dataset.tsv --output dataset.tsv
dataset = pandas.read_csv('dataset.tsv', sep='\t')
logging.info(f'Dataset contains {len(dataset)} rows\n')
dataset = dataset.sample(frac=1).reset_index(drop=True)
ipyplot.plot_images(
images=[row['url'] for _, row in dataset.iterrows()],
labels=[row['label'] for _, row in dataset.iterrows()],
max_images=12,
img_width=300,
)
```
--------------------------------
### Set up and Run Toloka Pipeline
Source: https://github.com/toloka/toloka-kit/blob/main/examples/3.audio_analysis/audio_transcription/audio_transcription.ipynb
Initializes an AssignmentsObserver to monitor submitted tasks and registers it with a Toloka Pipeline. It then opens the pool and starts the pipeline to process assignments.
```python
observer = AssignmentsObserver(toloka_client, pool_id=pool.id)
observer.on_submitted(handle_submitted)
pipeline = Pipeline()
pipeline.register(observer)
pool = toloka_client.open_pool(pool.id)
print(f'pool - {pool.status}')
```
--------------------------------
### Download Pipeline Script
Source: https://github.com/toloka/toloka-kit/blob/main/examples/metrics/jupyter_dashboard.ipynb
Downloads the 'find_items_pipeline.py' script required for the example. This is useful when running in environments like Google Colab.
```bash
!wget --quiet --show-progress "https://raw.githubusercontent.com/Toloka-kit/main/examples/metrics/find_items_pipeline.py"
```
--------------------------------
### Set Project Instructions
Source: https://github.com/toloka/toloka-kit/blob/main/examples/3.audio_analysis/audio_collection/audio_collection.ipynb
Defines the instructions for performers, guiding them on how to complete the audio recording tasks. It emphasizes recording each phrase separately.
```python
new_project.public_instructions = """Each task contains words and phrases. You need to read and record them.
Make a new recording for each phrase, even if it has already been used in other tasks."""
```
--------------------------------
### Async Iteration Example
Source: https://github.com/toloka/toloka-kit/blob/main/CHANGELOG.md
Demonstrates how to use async generators with the updated AsyncTolokaClient. This is the recommended way to iterate over results from asynchronous operations.
```python
async_toloka_client = AsyncTolokaClient(...)
async for item in async_toloka_client.get_tasks():
# process item
pass
```
--------------------------------
### Create Training Pool Configuration
Source: https://github.com/toloka/toloka-kit/blob/main/examples/autoquality/autoquality_usage.ipynb
Configures and creates a training pool for the project. This pool is used to train workers and is essential for AutoQuality setup.
```python
training_pool = toloka.training.Training(
project_id=project.id,
private_name='Training pool',
training_tasks_in_task_suite_count=5,
task_suites_required_to_pass=1,
may_contain_adult_content=False,
inherited_instructions=True,
assignment_max_duration_seconds=60*5,
retry_training_after_days=5,
mix_tasks_in_creation_order=True,
shuffle_tasks_in_task_suite=True,
)
training_pool = toloka_client.create_training(training_pool)
```
--------------------------------
### Initialize SentenceTransformer and TextHRRASA
Source: https://github.com/toloka/toloka-kit/blob/main/examples/3.audio_analysis/audio_transcription/audio_transcription.ipynb
Sets up the SentenceTransformer model for encoding and initializes the TextHRRASA object for audio transcription. Ensure the 'sentence-transformers' library is installed.
```python
from sentence_transformers import SentenceTransformer
from toloka.transformers.text.text_hrrasa import TextHRRASA
encoder = SentenceTransformer('paraphrase-distilroberta-base-v1')
hrrasa = TextHRRASA(lambda *args, **kwargs: encoder.encode(*args, show_progress_bar=False, **kwargs))
```
--------------------------------
### Setting up Toloka Pipeline with Observers
Source: https://github.com/toloka/toloka-kit/blob/main/examples/1.computer_vision/faces_detection/faces_detection.ipynb
Configures a pipeline with observers for detection and verification pools, utilizing local storage for persistence. This setup allows for restarting the pipeline without data loss.
```python
detection_observer = AssignmentsObserver(toloka_client, detection_pool.id)
detection_observer.on_submitted(DetectionSubmittedHandler(toloka_client, verification_pool.id))
verification_observer = AssignmentsObserver(toloka_client, verification_pool.id)
verification_observer.on_accepted(VerificationDoneHandler(toloka_client, verification_skill.id))
# Create a local directory that will store pipeline progress and logs.
# It allows to restart the pipeline without losing data in case of pause or failure.
storage_path = './storage/'
if not os.path.exists(storage_path):
os.makedirs(storage_path)
storage = JSONLocalStorage(storage_path)
pipeline = Pipeline(storage=storage)
pipeline.register(detection_observer)
pipeline.register(verification_observer)
# Google Colab is using a global event pool,
# so in order to run our pipeline we have to apply nest_asyncio to create an inner pool
if 'google.colab' in str(get_ipython()):
import nest_asyncio, asyncio
nest_asyncio.apply()
asyncio.get_event_loop().run_until_complete(pipeline.run())
else:
await pipeline.run()
```
--------------------------------
### Configure Project Interface and Plugins
Source: https://github.com/toloka/toloka-kit/blob/main/examples/autoquality/autoquality_usage.ipynb
Sets up the user interface for tasks, including view components like text viewers and radio buttons, and plugins for hotkeys and task layout.
```python
text_viewer = tb.TextViewV1(tb.InputData('text'))
radio_group_field = tb.ButtonRadioGroupFieldV1(
tb.OutputData('result'),
[
tb.GroupFieldOption('pos', '😃 Positive'),
tb.GroupFieldOption('neg', '😡 Negative'),
],
label='What is the review sentiment?',
validation=tb.RequiredConditionV1(hint='You need to select one answer'),
)
task_width_plugin = tb.TolokaPluginV1(
layout=tb.TolokaPluginV1.TolokaPluginLayout(
kind='pager',
task_width=500,
)
)
hot_keys_plugin = tb.HotkeysPluginV1(
key_1=tb.SetActionV1(tb.OutputData('result'), 'pos'),
key_2=tb.SetActionV1(tb.OutputData('result'), 'neg'),
)
project_interface = toloka.project.TemplateBuilderViewSpec(
view=tb.ListViewV1([radio_group_field, text_viewer]),
plugins=[task_width_plugin, hot_keys_plugin],
)
project.task_spec = toloka.project.task_spec.TaskSpec(
input_spec=input_specification,
output_spec=output_specification,
view_spec=project_interface,
)
```
--------------------------------
### Load Performer Instructions
Source: https://github.com/toloka/toloka-kit/blob/main/examples/2.spatial_crowdsourcing/0.simplest_example/spatial_crowdsourcing.ipynb
Loads prepared performer instructions from an HTML file. Ensure the 'instruction.html' file exists in the same directory or provide the correct path.
```python
prepared_instruction = open('instruction.html').read().strip()
```
--------------------------------
### Create Task Interface
Source: https://github.com/toloka/toloka-kit/blob/main/examples/3.audio_analysis/audio_classification/audio_classification.ipynb
Defines the user interface for tasks, including an audio player, radio buttons for selecting gender, and hotkeys for quick responses. This setup guides performers on how to interact with the task.
```python
audio_viewer = tb.AudioViewV1(
tb.InputData('path'),
validation=tb.PlayedConditionV1(hint='you need to listen to the audio'),
)
radio_group_field = tb.ButtonRadioGroupV1(
tb.OutputData('result'),
[
tb.GroupFieldOption('female', 'Female'),
tb.GroupFieldOption('male', 'Male'),
],
label='Is it a male or female speaker?',
validation=tb.RequiredConditionV1(),
)
task_width_plugin = tb.TolokaPluginV1(
layout=tb.TolokaPluginV1.TolokaPluginLayout(
kind='scroll',
task_width=300,
)
)
hot_keys_plugin = tb.HotkeysPluginV1(
key_1=tb.SetActionV1(tb.OutputData('result'), 'female'),
key_2=tb.SetActionV1(tb.OutputData('result'), 'male'),
)
project_interface = toloka.project.TemplateBuilderViewSpec(
view=tb.ListViewV1([audio_viewer, radio_group_field]),
plugins=[task_width_plugin, hot_keys_plugin],
)
```
--------------------------------
### Install Toloka-Kit Package
Source: https://github.com/toloka/toloka-kit/blob/main/README.md
Install the core Toloka-Kit package using pip. For production, specify the exact version. This command installs the core version by default; optional dependencies can be installed separately.
```bash
pip install toloka-kit
```
--------------------------------
### Create Project Configuration
Source: https://github.com/toloka/toloka-kit/blob/main/examples/8.search_relevance/search_relevance.ipynb
Defines the project's public name and description, which will be visible to performers.
```python
project = toloka.Project(
public_name='Classify search query relevance',
public_description='Analyze a website with a product and decide to what extent it meets the search query',
)
```
--------------------------------
### Install Dependencies
Source: https://github.com/toloka/toloka-kit/blob/main/examples/autoquality/autoquality_usage.ipynb
Installs the pandas library and the toloka-kit with autoquality support. Ensure you are using version 0.1.26 or compatible.
```python
!pip install pandas
!pip install toloka-kit[autoquality]==0.1.26
```
--------------------------------
### Write Project Instructions
Source: https://github.com/toloka/toloka-kit/blob/main/examples/5.nlp/sentiment_analysis/sentiment_analysis.ipynb
Defines the instructions for performers, explaining how to classify reviews as positive or negative and mentioning the available keyboard shortcuts. Clear instructions are crucial for accurate task completion.
```python
project.public_instructions = """In the task you will have to read customer reviews and define whether they are positive or negative
- Positive. Choose this option if the review reflects a customer's first-hand good experience with the product recommending to purchase it. For your convenience, you can also use the short-cut by pressing "1"
- Negative. Choose this option if the review reflects a customer's first-hand poor experience with the product recommending not to purchase it. For your convenience, you can also use the short-cut by pressing "2"
"""
```
--------------------------------
### Installing Toloka-Kit with Pandas Extra
Source: https://github.com/toloka/toloka-kit/blob/main/CHANGELOG.md
Install the 'pandas' extra to use methods like `TolokaClient.get_assignments_df`. This is required for DataFrame-based operations.
```bash
pip install toloka-kit[pandas]
```
--------------------------------
### Install Dependencies for Image Processing
Source: https://github.com/toloka/toloka-kit/blob/main/examples/faces_detection/faces_detection.ipynb
Installs necessary Python libraries for image manipulation and making HTTP requests. These are required for fetching and processing images.
```python
!pip install pillow # To deal with images
!pip install requests # To make HTTP requests
```
--------------------------------
### Set Up Verification Project Specifications
Source: https://github.com/toloka/toloka-kit/blob/main/examples/1.computer_vision/object_detection/object_detection.ipynb
Configures the Toloka project for verification, including its name, description, instructions, and task specifications for input, output, and view.
```python
verification_project = toloka.Project(
public_name='Are the traffic signs outlined correctly?',
public_description='Look at the image and decide whether or not the traffic signs are outlined correctly',
public_instructions=verification_instruction,
# Set up the task: view, input, and output parameters
task_spec=toloka.project.task_spec.TaskSpec(
input_spec={
'image': toloka.project.UrlSpec(),
'selection': toloka.project.JsonSpec(),
'assignment_id': toloka.project.StringSpec(),
},
# Set allowed_values, we'll use smart mixing to get the results of this project
output_spec={'result': toloka.project.StringSpec(allowed_values=['OK', 'BAD'])},
view_spec=verification_interface,
),
)
```
--------------------------------
### Create and Upload Toloka Project
Source: https://github.com/toloka/toloka-kit/blob/main/examples/2.spatial_crowdsourcing/0.simplest_example/spatial_crowdsourcing.ipynb
Creates a Toloka project instance with specified details and uploads it to Toloka. Requires `toloka` module and a `toloka_client` instance.
```python
project = toloka.Project(
public_name='Cleanliness of metro entrances',
public_description='Take two photos of the entrance on the right and on the left. The photo should show the entire entrances and the floor. So that you can assess the cleanliness around the entrance to the metro.',
public_instructions=prepared_instruction,
# We indicate that this task is selected by the performer on the map.
# Tasks without reference to the map are displayed simply as a list.
assignments_issuing_type='MAP_SELECTOR',
# We will indicate how the title of the task and description will be displayed on the map
assignments_issuing_view_config=toloka.Project.AssignmentsIssuingViewConfig(
title_template='Photo metro entrance', # Set title as a constant
description_template='{{inputParams["entrance"]}}', # Set description from the input parameters
# That way we can have different description for each point on the map
),
task_spec=toloka.project.task_spec.TaskSpec(
input_spec=input_specification,
output_spec=output_specification,
view_spec=project_interface
),
)
project = toloka_client.create_project(project)
```
--------------------------------
### Set Prefect Backend to Cloud
Source: https://github.com/toloka/toloka-kit/blob/main/examples/9.toloka_and_ml_on_prefect/example.ipynb
Configures Prefect to use its cloud-based service for orchestration. Ensure you have a Prefect Cloud account.
```bash
!prefect backend cloud
```
--------------------------------
### Create and Run Multiple Dashboards
Source: https://github.com/toloka/toloka-kit/blob/main/examples/metrics/jupyter_dashboard.ipynb
Demonstrates how to create and run multiple independent dashboard instances on different ports, useful for monitoring different requesters or projects simultaneously.
```python
dash_for_requester1 = DashBoard(toloka_client1, [...])
dash_for_requester2 = DashBoard(toloka_client2, [...])
```
```python
dash_for_requester1.run_dash(port='8081')
```
```python
dash_for_requester2.run_dash(port='8082')
```
--------------------------------
### Prepare and Create Tasks
Source: https://github.com/toloka/toloka-kit/blob/main/examples/benchmarks/image_classification_cinic10.ipynb
Generates a list of tasks for the training pool, where each task includes an image URL and its corresponding correct label. It uses known solutions to pre-define the expected output for each task.
```python
label_examples = {label: df[df.label == label].head(1).img_url.item() for label in CINIC_LABELS}
tasks = [
toloka.Task(input_values={'image': url},
known_solutions=[toloka.task.BaseTask.KnownSolution(output_values={'result': label})],
message_on_unknown_solution=f'Incorrect label! The actual label is: {label}',
infinite_overlap=True,
pool_id=training_pool.id)
for label, url in label_examples.items()
]
toloka_client.create_tasks(tasks, allow_defaults=True)
```
--------------------------------
### Setting up and Running the Faces Detection Pipeline
Source: https://github.com/toloka/toloka-kit/blob/main/examples/faces_detection/faces_detection.ipynb
Configures observers for detection and verification pools, initializes local storage, and registers observers with the pipeline. It handles asynchronous execution, adapting for environments like Google Colab.
```python
detection_observer = AssignmentsObserver(toloka_client, detection_pool.id)
detection_observer.on_submitted(DetectionSubmittedHandler(toloka_client, verification_pool.id))
verification_observer = AssignmentsObserver(toloka_client, verification_pool.id)
verification_observer.on_accepted(VerificationDoneHandler(toloka_client, verification_skill.id))
storage_path = './storage/'
if not os.path.exists(storage_path):
os.makedirs(storage_path)
storage = JSONLocalStorage(storage_path)
pipeline = Pipeline(storage=storage)
pipeline.register(detection_observer)
pipeline.register(verification_observer)
if 'google.colab' in str(get_ipython()):
import nest_asyncio, asyncio
nest_asyncio.apply()
asyncio.get_event_loop().run_until_complete(pipeline.run())
else:
await pipeline.run()
```
--------------------------------
### Sample Misclassified Examples
Source: https://github.com/toloka/toloka-kit/blob/main/examples/benchmarks/text_classification_imdb.ipynb
Filters a DataFrame to show examples where the predicted label does not match the true label, then samples and displays the first 5 of these misclassified text entries along with their predicted and true labels. Useful for qualitative analysis of model errors.
```python
aggregated_answers[aggregated_answers.pred_label != aggregated_answers.label][['text', 'pred_label', 'label']].sample(5).head()
```
--------------------------------
### Toloka Marking Project Interface Setup
Source: https://github.com/toloka/toloka-kit/blob/main/examples/SQUAD2.0/SQUAD2.0_processing.ipynb
This Python code defines the interface for a Toloka marking project. It includes fields for text, question, and a radio group to determine if an answer is present, with a conditional textarea for the answer itself. This setup is used for tasks where performers identify answers to questions within a given text.
```python
# How performers will see the task
radio_group_field = tb.RadioGroupFieldV1(
tb.OutputData('is_possible'),
[
tb.GroupFieldOption('yes', 'Yes'),
tb.GroupFieldOption('no', 'No')
],
label='Does the text contain an answer?',
validation=tb.RequiredConditionV1()
)
helper = tb.helpers.IfHelperV1(
tb.EqualsConditionV1(
'yes',
tb.OutputData('is_possible')
),
tb.TextareaFieldV1(
tb.OutputData('answer'),
label='Paste an answer',
validation=tb.RequiredConditionV1()
)
)
project_interface = toloka.project.TemplateBuilderViewSpec(
view=tb.ListViewV1(
[
tb.TextViewV1(tb.InputData('text'), label='Text'),
tb.TextViewV1(tb.InputData('question'), label='Question'),
tb.ListViewV1([radio_group_field, helper])
]
)
)
public_instruction = open('marking_public_instruction.html').read().strip()
```
--------------------------------
### Prepare Training Dataset with Hints
Source: https://github.com/toloka/toloka-kit/blob/main/examples/5.nlp/intent_classification/intent_classification.ipynb
Creates a Pandas DataFrame for training tasks, including queries, their correct domain and intent, and a descriptive hint explaining why the answer is correct. This dataset is used to train or fine-tune classification models.
```python
training_data = [
['are there a reservation available at xenophone',
'kitchen',
'restaurant_reservation',
'It is reservation restaurant class in kitchen category. Try to find key words: reserve / book , name of the restaurant'],
['I need a table for 7 pm under the name Paul',
'kitchen',
'restaurant_reservation',
'Query can consist of key words like table, time, name. Category kitchen, class reservation'],
['what are the review for burger king',
'kitchen',
'restaurant_reviews',
'Pay attention to existence of the name of restaurant or the review key word. Category kitchen, class review'],
['can I sub ketchup for mayo',
'kitchen',
'recipe',
'Anything related to substitution. Key words: instead, substitution, sub. Category kitchen, class recipe'],
['pull up soup recipe',
'kitchen',
'recipe',
"One of common words for such class could be 'recipe'. Category kitchen, class recipe"],
['How healthy is orange',
'kitchen',
'nutrition_info',
"'healthy' word is a marker of nutrition class. Category kitchen, class nutrition"],
['How many calories are in mushrooms',
'kitchen',
'nutrition_info',
'Select this class if something is said about the amount of calories. Category kitchen, class nutrition'],
['in which time zone does Denver reside',
'travel',
'timezone',
'You can notice basic words like zone and time. As a result, category is travel and class is timezone'],
['find me a hotel with good reviews in phoenix',
'travel',
'book_hotel',
'If you find the word hotel than it could be a sign of category travel and class book a hotel'],
["what's the best place for travelling to this time of year",
'travel',
'travel_suggestion',
'Intent which consists of any recommendations about traveling. Category travel, class travel suggestion'],
['find me the exchange rate between usd and cad',
'travel',
'exchange_rate',
'Type of currency written in the query suggests category travel with class exchange rate'],
['are more shots needed to travel to argentina',
'travel',
'vaccines',
'Intent with shots and vaccines - key words related to category travel and class vaccine']
]
training_dataset = pd.DataFrame(training_data,
columns=['query', 'domain', 'intent', 'hint']
)
```
--------------------------------
### Define Input and Output Specifications
Source: https://github.com/toloka/toloka-kit/blob/main/examples/3.audio_analysis/audio_classification/audio_classification.ipynb
Sets up the data specifications for the project, defining the input field 'path' as a URL and the output field 'result' as a string. This ensures data is correctly formatted and processed.
```python
input_specification = {'path': toloka.project.UrlSpec()}
output_specification = {'result': toloka.project.StringSpec()}
project.task_spec = toloka.project.task_spec.TaskSpec(
input_spec=input_specification,
output_spec=output_specification,
view_spec=project_interface,
)
```
--------------------------------
### Get Best Pool ID
Source: https://github.com/toloka/toloka-kit/blob/main/examples/autoquality/autoquality_usage.ipynb
Access the `best_pool_id` attribute after running AutoQuality to find the ID of the most optimal pool identified.
```python
aq.best_pool_id
```
--------------------------------
### Configure Task Interface with Template Builder
Source: https://github.com/toloka/toloka-kit/blob/main/examples/3.audio_analysis/audio_collection/audio_collection.ipynb
Designs the user interface for the task, including a text view for displaying text to be read and an audio field for recording. It also includes a plugin for task width.
```python
text_view = tb.TextViewV1(tb.InputData('text'))
audio_field = tb.AudioFieldV1(tb.OutputData('audio_file'), validation=tb.RequiredConditionV1())
width_plugin = tb.TolokaPluginV1('scroll', task_width=500)
project_interface = toloka.project.TemplateBuilderViewSpec(
view=tb.ListViewV1(items=[text_view, audio_field]),
plugins=[width_plugin]
)
```
--------------------------------
### Create a Training Pool
Source: https://github.com/toloka/toloka-kit/blob/main/examples/5.nlp/text_classification/text_classification.ipynb
Initializes and creates a training pool for a project. Configure parameters like task duration, task suite counts, and retry intervals.
```python
training = toloka.Training(
project_id=project.id,
private_name='clickbait training',
may_contain_adult_content=False,
assignment_max_duration_seconds=60*30,
mix_tasks_in_creation_order=False,
shuffle_tasks_in_task_suite=False,
training_tasks_in_task_suite_count=10,
task_suites_required_to_pass=10,
retry_training_after_days=10,
inherited_instructions=True,
)
training = toloka_client.create_training(training)
```
--------------------------------
### Define Verification Project Instructions
Source: https://github.com/toloka/toloka-kit/blob/main/examples/1.computer_vision/object_detection/object_detection.ipynb
Sets the HTML instructions for the verification task, guiding performers on how to assess the outlined traffic signs.
```html
verification_instruction = '''Look at the image and answer the question:
Are all traffic signs outlined correctly?
If they are, click Yes.
If they aren't, click No.
For example, the road signs here are outlined correctly, so the correct answer is Yes.'''
```
--------------------------------
### Create a New Toloka Project
Source: https://github.com/toloka/toloka-kit/blob/main/examples/1.computer_vision/text_recognition/text_recognition.ipynb
Initializes a new Toloka project with a public name and description. This sets the basic metadata for your project.
```python
project = toloka.Project(
public_name='Write down the digits in an image',
public_description='Look at the image and write down the digits shown on the water meter.',
)
```
--------------------------------
### Set Project Instructions
Source: https://github.com/toloka/toloka-kit/blob/main/examples/3.audio_analysis/audio_classification/audio_classification.ipynb
Defines the public instructions for performers, guiding them to listen to audio clips and determine the speaker's gender.
```python
project.public_instructions = """Listen to the short audio clip and determine whether it is a male or female speaking."""
```
--------------------------------
### Initialize Pipeline
Source: https://github.com/toloka/toloka-kit/blob/main/examples/metrics/jupyter_dashboard.ipynb
Initializes the FindItemsPipeline using the configured Toloka client. This step prepares the pipeline for execution.
```python
from find_items_pipeline import FindItemsPipeline
pipeline = FindItemsPipeline(client=toloka_client)
pipeline.init_pipeline()
```
--------------------------------
### Getting Specific Toloka Operations
Source: https://github.com/toloka/toloka-kit/blob/main/CHANGELOG.md
Use `TolokaClient.get_operations` to retrieve a list of operations. This method complements `find_operations` for managing operation status.
```python
toloka_client.get_operations(...)
```
--------------------------------
### Import Libraries
Source: https://github.com/toloka/toloka-kit/blob/main/examples/benchmarks/text_classification_imdb.ipynb
Imports essential Python libraries for data manipulation, machine learning, and Toloka client interaction. Ensure these are installed before running.
```python
import datetime
import time
import pandas as pd
import numpy as np
from sklearn.metrics import balanced_accuracy_score
import os
import logging
import sys
import getpass
import requests
from tqdm.auto import tqdm
import toloka.client as toloka
import toloka.client.project.template_builder as tb
from crowdkit.aggregation import DawidSkene
%matplotlib inline
pd.options.display.max_colwidth = 300
```
--------------------------------
### Instantiate and Initialize Pipeline
Source: https://github.com/toloka/toloka-kit/blob/main/examples/metrics/graphite.ipynb
Imports the FindItemsPipeline class and creates an instance of the pipeline using the initialized Toloka client. The pipeline is then initialized.
```python
from find_items_pipeline import FindItemsPipeline
pipeline = FindItemsPipeline(client=toloka_client)
pipeline.init_pipeline()
```
--------------------------------
### Project Interface Configuration
Source: https://github.com/toloka/toloka-kit/blob/main/examples/2.spatial_crowdsourcing/0.simplest_example/spatial_crowdsourcing.ipynb
Sets up the project's interface view, including layout, validation rules, and plugins. Requires `toloka.project.TemplateBuilderViewSpec` and `toloka.tb` modules.
```python
# How performers will see the task
project_interface = toloka.project.TemplateBuilderViewSpec(
view=tb.ListViewV1(
[header, entrance_name, workflow_options, exist_ui, miss_ui],
validation=coordinates_validation,
),
plugins=[task_width_plugin, coordinates_save_plugin]
)
```
--------------------------------
### Set Up Verification Project
Source: https://github.com/toloka/toloka-kit/blob/main/examples/1.computer_vision/faces_detection/faces_detection.ipynb
Creates a Toloka project for verifying face annotations. It defines the project's public details, input specifications (image, selection, assignment ID), output specifications (result: OK/BAD), and the task view.
```python
verification_instruction = open('./instructions/verification_instruction.html').read().strip()
# Set up the project
verification_project = toloka.Project(
public_name='Are the people faces outlined correctly?',
public_description='Look at the image and decide whether or not the people faces are outlined correctly',
public_instructions=verification_instruction,
# Set up the task: view, input, and output parameters
task_spec=toloka.project.task_spec.TaskSpec(
input_spec={
'image': toloka.project.UrlSpec(),
'selection': toloka.project.JsonSpec(),
'assignment_id': toloka.project.StringSpec(),
},
# Set allowed_values, we'll use smart mixing to get the results of this project
output_spec={'result': toloka.project.StringSpec(allowed_values=['OK', 'BAD'])},
view_spec=verification_interface,
),
)
```
--------------------------------
### Graphite Count Metric Aggregation
Source: https://github.com/toloka/toloka-kit/blob/main/examples/metrics/graphite.ipynb
Configures Graphite to use the 'sum' aggregation method for metrics ending with '.count'. This is a common default setup.
```ini
[count]
pattern = \.count$
xFilesFactor = 0
aggregationMethod = sum
```
--------------------------------
### Getting a Specific Toloker User
Source: https://github.com/toloka/toloka-kit/blob/main/CHANGELOG.md
Use `TolokaClient.get_user` to retrieve information about a single Toloker. This method is useful for fetching details about a specific user.
```python
toloka_client.get_user(user_id)
```
--------------------------------
### Initialize Task List
Source: https://github.com/toloka/toloka-kit/blob/main/examples/autoquality/autoquality_usage.ipynb
Initialize an empty list to store the tasks that will be created for AutoQuality.
```python
aq_tasks = []
```
--------------------------------
### Import Required Libraries
Source: https://github.com/toloka/toloka-kit/blob/main/examples/5.nlp/intent_classification/intent_classification.ipynb
Imports all essential modules for Toloka client operations, template building, data handling, and logging. These are needed for most Toloka-Kit examples.
```python
import datetime
import json
import logging
import sys
import time
import pandas as pd
import toloka.client as toloka
import toloka.client.project.template_builder as tb
from crowdkit.aggregation import DawidSkene
```
--------------------------------
### Open Toloka Pools for Tasks
Source: https://github.com/toloka/toloka-kit/blob/main/examples/8.search_relevance/search_relevance.ipynb
Opens training, exam, and main pools to start accepting performer responses. It prints the status of each pool after opening.
```python
training = toloka_client.open_training(training.id)
print(f'training - {training.status}')
exam = toloka_client.open_pool(exam.id)
print(f'exam - {exam.status}')
pool = toloka_client.open_pool(pool.id)
print(f'main pool - {pool.status}')
```