### Install PuLP Library

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/markov-chain-lpp.ipynb

Installs the PuLP library, a Python modeling language for optimization problems, which is required for setting up and solving the linear programming problem.

```bash
!pip install pulp
```

--------------------------------

### LightGBM Data Preparation and Cross-Validation Setup

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/desafio_valorizacao/Descobrindo o algoritmo de valorização do Cartola FC - Parte II.ipynb

Prepares data for LightGBM training, defining feature names, categorical features, and creating LightGBM Dataset objects for training and testing. Sets up parameters for regression objective and mean squared error metric.

```python
# LightGBM
features_names = ['rodada', 'preco', 'media_pontos', 
                  'pontos_lag', 'jogou_partida', 'posicao_rec',
                  'med_preco', 'med_pontos_lag']

train_data = lgb.Dataset(train_features, label=train_result,
                        feature_name=features_names,
                        categorical_feature=['jogou_partida','posicao_rec'],
                        free_raw_data=False)

test_data = lgb.Dataset('test.svm', reference=train_data)

params = {
    'objective':'regression',
    'metric':'mean_squared_error',
}

# Training
num_round = 500
cv_results = lgb.cv(params, train_data, num_round, 
                    nfold=10,
                    stratified=False,
                    verbose_eval=20,
                    early_stopping_rounds=40)
```

--------------------------------

### Initializing the Linear Programming Problem

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/markov-chain-lpp.ipynb

Sets up a linear programming problem to maximize the objective function, defining decision variables for player selection. This is the core of the optimization setup.

```python
prob = LpProblem("Melhor_Escalacao", LpMaximize)
y = LpVariable.dicts("Atl",df_rodada.index,0,1,cat='Binary')
prob += lpSum([z[i] * y[i] for i in y])
```

--------------------------------

### Import Python Libraries

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/colabs/caRtola_como_ler_repositório_do_github_com_BeautifulSoup_e_Pandas.ipynb

Imports essential libraries for web scraping and data manipulation. Ensure these libraries are installed in your environment.

```python
# Importar bibliotecas
import re                          # Expressão regulares
import requests                    # Acessar páginas da internet
from bs4 import BeautifulSoup      # Raspar elementos de páginas da internet
import pandas as pd                # Abrir e concatenar bancos de dados
```

--------------------------------

### Create Conda Environment

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/README.md

Use this command to create a new Conda environment based on the provided 'cartola.yml' file. This ensures all necessary Python dependencies are installed.

```sh
conda env create -f cartola.yml
```

--------------------------------

### Load and Inspect Data

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/Análise dos Dados.ipynb

Loads a CSV file into a pandas DataFrame and prints its shape. Use this to quickly load and get an overview of your dataset.

```python
df_samples = pd.read_csv('../../data/dados_agregados_limpos.csv')
print(df_samples.shape)
df_samples.head()
```

--------------------------------

### Display Initial Transition Matrix

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/markov-chain-lpp.ipynb

Prints the initial state of the transition matrix 'M', which is filled with zeros before any game data is processed.

```python
M
```

--------------------------------

### Get shape of combined round data

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/markov-chain-lpp.ipynb

Returns the dimensions of the 'df_partidas' DataFrame, showing the total number of player performances across all rounds.

```python
df_partidas.shape
```

--------------------------------

### Get shape of player data DataFrame

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/markov-chain-lpp.ipynb

Returns the dimensions (number of rows and columns) of the 'medias' DataFrame. Useful for understanding the dataset size.

```python
medias.shape
```

--------------------------------

### Initialize Linear Programming Problem

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/markov-chain-lpp.ipynb

Imports necessary PuLP functions, initializes a maximization problem named 'Melhor_Escalacao', and defines binary variables for each player.

```python
from pulp import LpMaximize, LpProblem, lpSum, LpVariable

prob = LpProblem("Melhor_Escalacao", LpMaximize)
y = LpVariable.dicts("Atl",df_rodada.index,0,1,cat='Binary')
prob += lpSum([z[i] * y[i] for i in y])
```

--------------------------------

### Display First Four Dictionary Entries

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/colabs/caRtola_como_ler_repositório_do_github_com_BeautifulSoup_e_Pandas.ipynb

Prints the first four key-value pairs from the created dictionary to show the extracted file names and their corresponding raw URLs.

```python
# Imprimir os primeiros casos do dicionário criado
dict(list(dict_of_files.items())[0:4]) # Ignore este código horrível
```

--------------------------------

### Get shape of match data DataFrame

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/markov-chain-lpp.ipynb

Returns the dimensions (number of rows and columns) of the 'partidas' DataFrame. Shows the total number of matches recorded.

```python
partidas.shape
```

--------------------------------

### Get shape of unique player data DataFrame

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/markov-chain-lpp.ipynb

Returns the dimensions (number of rows and columns) of the 'atletas' DataFrame. Confirms the number of unique players.

```python
atletas.shape
```

--------------------------------

### Load and Inspect Initial Data

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/desafio_valorizacao/Desafio da Valorização.ipynb

Loads the dataset from a CSV file and prints its shape. Displays the first few rows to give an overview of the data.

```python
df = pd.read_csv('../../../data/desafio_valorizacao/valorizacao_cartola_2018.csv')
print(df.shape)

df.head()
```

--------------------------------

### Create Training Samples by Merging Rounds

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/desafio_valorizacao/Desafio da Valorização.ipynb

Iterates through each round to create training samples. It merges the current round's data with the next round's price variation for the same player, dropping rows with missing values or zero price variation in either round. This process is timed for performance analysis.

```python
%%time
df_samples = pd.DataFrame([])

for rodada in range(1, 38):
    df_rod_atual = df[df.rodada == rodada]
    df_rod_prox = df[df.rodada == (rodada + 1)]
    df_merge = df_rod_atual.merge(df_rod_prox[['id', 'variacao_preco']] , how='left', on='id', suffixes=('_atual', '_prox'))
    df_merge = df_merge.dropna()
    df_merge = df_merge[(df_merge.variacao_preco_atual != 0) & (df_merge.variacao_preco_prox != 0)]
    df_samples = df_samples.append(df_merge)
    
print(df_samples.shape)
```

--------------------------------

### Get unique player positions

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/markov-chain-lpp.ipynb

Extracts all unique player positions from the 'player_position' column in the 'medias' DataFrame. This list is used for separate analysis per position.

```python
posicoes = medias['player_position'].unique()
```

--------------------------------

### Load and Display Sample Data

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/Análise dos Dados.ipynb

Loads and displays the first few rows of a pandas DataFrame containing player statistics. Assumes 'df_samples' is already loaded.

```python
cols_info = ['Rodada', 'ano']

df_samples = df_samples[cols_of_interest]
df_samples.head()
```

--------------------------------

### Displaying Sample Data

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/Análise dos Dados.ipynb

Shows a sample of the prepared data, including various features and predictions. This is useful for understanding the structure and content of the dataset.

```python
4         0.4        1.52           0.0  18.0          0.0          0.0   

   home.score.x  mes  pred.away.score  pred.home.score  risk_points   variable  
0           1.0  4.0              0.0              0.0          1.0  home.team  
1           2.0  4.0              0.0              0.0          1.0  away.team  
2           2.0  5.0              0.0              0.0          1.0  home.team  
3           2.0  5.0              0.0              0.0          1.0  home.team  
4           0.0  5.0              0.0              0.0          1.0  away.team  
```

--------------------------------

### Filter Top 10 Athletes and Get IDs

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/colabs/caRtola_media_media_movel_media_casa_ou_fora_o_que_usar.ipynb

Filters the DataFrame to select the top 10 athletes from the 2019 season based on average score, excluding technical coaches. It then extracts the athlete IDs.

```python
# Selecionar top 10 atletas de 2019
top_df = df[(df['atletas.rodada_id'] == 38) & 
            (df['atletas.posicao_id'] != 'tec') & 
            (df['atletas.atleta_id']).isin(dict_of_players_more_than_18.keys())]

# Criar uma lista dos 10 melhores jogadores
top_players = top_df.sort_values('atletas.media_num', ascending=False).head(10)
top_players = top_players['atletas.atleta_id'].tolist()
```

--------------------------------

### Team Selection with Markov Chains and Linear Programming in Python

Source: https://context7.com/henriquepgomide/cartola/llms.txt

Combines Markov Chain stationary distributions for player ranking with Integer Linear Programming (PuLP) for optimal team formation, respecting formation and budget constraints. Requires pandas, numpy, and pulp libraries.

```python
# notebooks/markov-chain-lpp.ipynb — versão condensada
import pandas as pd
import numpy as np
from pulp import LpMaximize, LpProblem, lpSum, LpVariable

# 1. Carrega médias e partidas de 2019
medias   = pd.read_csv("data/01_raw/2019/2019-medias-jogadores.csv")
partidas = pd.read_csv("data/01_raw/2019/2019_partidas.csv")
partidas["home_score_norm"] = partidas["home_score"] / partidas["home_score"].max()
partidas["away_score_norm"] = partidas["away_score"] / partidas["away_score"].max()

# 2. Constrói e normaliza matriz de transições por posição
def build_markov_matrix(df_partidas_posicao, partidas, atletas, posicao):
    qtd = len(atletas[atletas.player_position == posicao])
    M   = np.zeros((qtd, qtd))
    for i in range(len(partidas) - 1):
        rodada       = df_partidas_posicao[df_partidas_posicao["round"] == partidas["round"][i]]
        j_casa       = rodada[rodada["atletas.clube_id"] == partidas["home_team"][i]]
        j_visitantes = rodada[rodada["atletas.clube_id"] == partidas["away_team"][i]]
        for j1_row in j_casa.itertuples():
            for j2_row in j_visitantes.itertuples():
                soma = j1_row.atletas_pontos_num + j2_row.atletas_pontos_num
                sc   = (j1_row.atletas_pontos_num / soma) if soma != 0 else 0
                M[j1_row.Rank, j1_row.Rank] += partidas["home_score_norm"][i] + sc
                M[j2_row.Rank, j2_row.Rank] += partidas["away_score_norm"][i] + (1 - sc)
    return M / np.sum(M, axis=1, keepdims=True)

# 3. Distribuição estacionária via autovetor
def stationary_dist(M):
    evals, evecs = np.linalg.eig(M.T)
    evec = evecs[:, np.isclose(evals, 1)][:, 0]
    return (evec / evec.sum()).real

# 4. Programação Linear — formação 4-3-3, orçamento 140 cartoletas
formacao   = {"ata": 3, "mei": 3, "lat": 2, "zag": 2, "gol": 1}
cartoletas = 140

df_rodada = df_partidas[df_partidas["round"] == 38].copy()
# ... (atribui probs da distribuição estacionária ao df_rodada)

z = df_rodada["probs"].to_dict()
c = df_rodada["atletas.preco_num"].to_dict()
dummies = pd.get_dummies(df_rodada["atletas.posicao_id"]).to_dict()

prob = LpProblem("Melhor_Escalacao", LpMaximize)
y    = LpVariable.dicts("Atl", df_rodada.index, 0, 1, cat="Binary")
prob += lpSum([z[i] * y[i] for i in y])
prob += lpSum([c[i] * y[i] for i in y]) <= cartoletas, "Orcamento"
for pos, qtd in formacao.items():
    prob += lpSum([dummies[pos][i] * y[i] for i in y]) == qtd, f"Qtd_{pos}"

prob.solve()

escalados = [v.name.replace("Atl_","").replace("_","-")
             for v in prob.variables() if v.varValue == 1]
print(df_rodada.loc[escalados][["atletas.posicao_id","atletas.pontos_num","atletas.preco_num"]])
# Total de pontos na rodada 38: 81.4 | Custo: 118.9 cartoletas
```

--------------------------------

### Prepare Data for Linear Programming

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/markov-chain-lpp.ipynb

Sets the player slug as the index, converts probabilities and costs to dictionaries, and creates dummy variables for player positions to be used in the optimization problem.

```python
df_rodada.set_index('atletas.slug',inplace=True)
z = df_rodada['probs'].to_dict()
c = df_rodada['atletas.preco_num'].to_dict()

dummies_posicao = pd.get_dummies(df_rodada['atletas.posicao_id'])
dummies_posicao = dummies_posicao.to_dict()
```

--------------------------------

### Load and Inspect Team Data

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/Análise dos Dados.ipynb

Reads team data from a CSV file, removes rows with missing values, and prints the dimensions of the resulting DataFrame. Use this to load and get an initial overview of the team dataset.

```python
df_teams = pd.read_csv('../../data/times_ids.csv')
df_teams = df_teams.dropna()
print(df_teams.shape)
df_teams.head()
```

--------------------------------

### Loading Model and Preparing Test Data

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/Análise dos Dados.ipynb

Loads a pre-trained model from a pickle file and reads test data for predictions. Filters data for the year 2017.

```python
df_test = pd.read_csv('../../data/dados_agregados_limpos.csv')
df_test = df_test[df_test.ano == 2017]
reg = pkl.load(open('../../src/python/models/nn1.pkl', 'rb'))
```

--------------------------------

### Define Team Formation and Budget

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/markov-chain-lpp.ipynb

Sets the desired number of players for each position and the total budget ('cartoletas') available for team selection.

```python
formacao = {
    'ata': 3,
    'mei': 3,
    'lat': 2,
    'zag': 2,
    'gol':1
}

cartoletas = 140
```

--------------------------------

### Prepare Round Data and Initialize Probabilities

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/markov-chain-lpp.ipynb

Filters the main dataset for a specific round and initializes a 'probs' column to zero for each player. Ensures the 'Rank' column is of integer type.

```python
df_rodada = df_partidas[df_partidas['round'] == rodada].copy()
df_rodada['Rank'] = df_rodada['Rank'].astype(int)
df_rodada['probs'] = 0
```

--------------------------------

### Import Libraries and Set Options

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/desafio_valorizacao/Desafio da Valorização.ipynb

Imports necessary libraries like pandas, numpy, and matplotlib, and sets pandas display options. Also includes a magic command for inline plotting.

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score

pd.set_option('max_columns', 100)

%matplotlib inline
```

--------------------------------

### Create Training Samples with Player Data

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/Análise dos Dados.ipynb

Generates a dataframe containing player scouts from a specified training round and their corresponding scores from a prediction round. Requires a pandas DataFrame with player and round information.

```python
def create_samples(df, round_train, round_pred):
    '''Create a Dataframe with players from round_train, but with 'Pontos' of round_pred'''
    df_train = df[df['Rodada'] == round_train]
    df_pred = df[df['Rodada'] == round_pred][['AtletaID', 'Pontos']]
    df_merge = df_train.merge(df_pred, on='AtletaID', suffixes=['_train', '_pred'])
    
    df_merge = df_merge.rename(columns={'Pontos_train':'Pontos', 'Pontos_pred':'pred'})
    
    return df_merge
```

--------------------------------

### Prepare Match Data for Analysis (Python)

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/colabs/caRtola_media_media_movel_media_casa_ou_fora_o_que_usar.ipynb

Prepares the 2019 match data for further analysis by selecting relevant columns, melting the DataFrame to distinguish home and away teams, and renaming columns for consistency. This step transforms the data to facilitate analysis of home and away performance.

```python
# Preparar banco de dados
partidas_2019 = partidas_2019[['home_team', 'away_team', 'round']]
partidas_2019 = partidas_2019.melt(id_vars='round', value_name='atletas.clube_id')
partidas_2019 = partidas_2019.rename(columns={'round': 'atletas.rodada_id', 
                                              'variable': 'home_away'})
```

--------------------------------

### Create and populate player rank column

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/markov-chain-lpp.ipynb

Initializes a 'Rank' column with None and then calculates a rank for each player within their respective positions based on 'player_id'. This rank is crucial for indexing players in position-specific matrices.

```python
medias['Rank'] = None
for posicao in posicoes:
    rank = medias[medias['player_position'] == posicao].player_id.rank(method='min')
    rank = rank - 1
    medias.iloc[rank.index,-1] = rank
```

--------------------------------

### Fit Linear Regression Model and Predict Player Price Variation

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/desafio_valorizacao/Descobrindo o algoritmo de valorização do Cartola FC - Parte I.ipynb

Prepares data by filtering and dropping nulls, then fits a linear regression model to predict price variation based on historical performance and price. It then prints the model's intercept, coefficients, and evaluation metrics.

```python
paqueta_complete = paqueta[(~paqueta.status.isin(['Nulo', 'Suspenso'])) & (paqueta.rodada > 5)]
paqueta_complete = paqueta_complete.dropna()

predictors = paqueta_complete[['pontos_lag','preco','media_lag']]
outcome = paqueta_complete['variacao_preco_lag']

regr = linear_model.LinearRegression()
regr.fit(predictors, outcome)
paqueta_complete['predictions'] = regr.predict(paqueta_complete[['pontos_lag', 'preco', 'media_lag']])

print('Intercept: \n', regr.intercept_)
print('Coefficients: \n', regr.coef_)
print("Mean squared error: %.2f"
      % mean_squared_error(paqueta_complete['variacao_preco_lag'], paqueta_complete['predictions']))
print('Variance score: %.2f' % r2_score(paqueta_complete['variacao_preco_lag'], paqueta_complete['predictions']))
```

--------------------------------

### Display first 5 rows of player data

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/markov-chain-lpp.ipynb

Shows the first five rows of the 'medias' DataFrame to give a quick overview of the loaded player data.

```python
medias.head()
```

--------------------------------

### Exibir as Primeiras Linhas do DataFrame de Jogadores

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/colabs/caRtola_media_media_movel_media_casa_ou_fora_o_que_usar.ipynb

Mostra as primeiras 5 linhas do DataFrame `df` após o carregamento e pré-processamento, permitindo uma visualização inicial dos dados dos jogadores.

```python
# Ver início do banco de dados
df.head()
```

--------------------------------

### Data Sample Preparation Function

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/Análise dos Dados.ipynb

Prepares a DataFrame into a format suitable for model input by mapping categorical features to integers and selecting relevant columns. Assumes 'cols_info' and 'cols_of_interest' are defined.

```python
def to_samples(df):
    df_samples = df[cols_info+cols_of_interest].copy()
    df_samples['ClubeID'] = df_samples['ClubeID'].map(dict_teams(to_int=True)) 
    df_samples['Posicao'] = df_samples['Posicao'].map(dict_positions(to_int=True))
    df_samples['variable'] = df_samples['variable'].map({'home.team':1, 'away.team':2})
    df_samples.reset_index(drop=True, inplace=True)
    return df_samples
```

--------------------------------

### Initialize Transition Matrix

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/markov-chain-lpp.ipynb

Initializes a square transition matrix 'M' with zeros. The size of the matrix is determined by the number of athletes in the specified position ('ata').

```python
qtd_atletas = len(atletas[atletas.player_position == posicao])
M = np.zeros((qtd_atletas,qtd_atletas))
```

--------------------------------

### Prepare Data for Player Prediction

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/Análise dos Dados.ipynb

Filters the test dataset for a specific round and player status ('Provável'), then converts it into a sample format for prediction. Prints the shape of the generated samples.

```python
df_rodada = df_test[(df_test['Rodada'] == (ROUND_TO_PREDICT-1)) & (df_test['Status'] == "Provável")]
df_samples = to_samples(df_rodada)
print(df_samples.shape)
```

--------------------------------

### Import Libraries and Load Data

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/desafio_valorizacao/Descobrindo o algoritmo de valorização do Cartola FC - Parte I.ipynb

Imports necessary Python libraries for data analysis and machine learning, sets Pandas display options, and loads the 2018 Cartola FC valuation data from a CSV file.

```python
# Importar bibliotecas
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn import linear_model
from sklearn.metrics import mean_squared_error, r2_score

pd.options.mode.chained_assignment = None  # default='warn'
%matplotlib inline
pd.options.display.float_format = '{:,.2f}'.format

# Abrir banco de dados
dados = pd.read_csv('~/caRtola/data/desafio_valorizacao/valorizacao_cartola_2018.csv')
```

--------------------------------

### Team Selection with Linear Programming in R

Source: https://context7.com/henriquepgomide/cartola/llms.txt

Implements team selection optimization in R using the lpSolve package. It maximizes predicted scores from an XGBoost model while adhering to positional quotas and budget constraints. Requires the lpSolve library and a dataframe 'df_pred_r2' with predictions.

```r
# src/R/player_analysis/select_team.R
library(lpSolve)

# df_pred_r2: jogadores prováveis da próxima rodada com coluna next_round (previsão)
matrix <- rbind(
  as.numeric(df_pred_r2$Posicao == "ata"), as.numeric(df_pred_r2$Posicao == "ata"),
  as.numeric(df_pred_r2$Posicao == "mei"), as.numeric(df_pred_r2$Posicao == "mei"),
  as.numeric(df_pred_r2$Posicao == "zag"), as.numeric(df_pred_r2$Posicao == "zag"),
  as.numeric(df_pred_r2$Posicao == "lat"), as.numeric(df_pred_r2$Posicao == "lat"),
  as.numeric(df_pred_r2$Posicao == "gol"),
  as.numeric(df_pred_r2$Posicao == "tec"),
  diag(df_pred_r2$risk_points),  # cada jogador pode ter no máx. 5 pts de risco
  df_pred_r2$Preco               # restrição de orçamento total
)

sol <- lp(
  direction    = "max",
  objective.in = df_pred_r2$next_round,
  const.mat    = matrix,
  const.dir    = c("<=",">=", "<=",">=", "<=",">=", "<=",">=",
                   "==","==", rep("<=", nrow(df_pred_r2)), "<="),
  const.rhs    = c(3,1, 5,3, 3,2, 2,0, 1, 1,
                   rep(5, nrow(df_pred_r2)), 97.11),
  all.bin      = TRUE
)

# Elenco ótimo selecionado:
inds <- which(sol$solution == 1)
df_pred_r2[inds, c("Apelido","Posicao","next_round","Preco")]
```

--------------------------------

### Run Jupyter Notebook

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/README.md

Launch the Jupyter Notebook interface from your terminal. This allows you to interactively run and explore the Python notebooks in the repository.

```sh
jupyter notebook
```

--------------------------------

### Create Training Samples for LSTM

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/LSTM.ipynb

Prepares data samples for training an LSTM model. It creates input sequences (x) and corresponding target values (y) from a DataFrame, considering specified rounds and historical data.

```python
def create_samples(df, cols_of_interest, from_round=1, to_round=None, last_rounds=None, return_pred=True):
    players = list(df[COL_ID].unique())
    rounds = list(df[COL_ROUND].unique())
    n_rounds = max(rounds)
    n_players = len(players)
    to_round = n_rounds if to_round is None else to_round
    last_rounds = n_rounds - 1 if last_rounds is None else last_rounds
    
    list_rounds = np.array([list(range(1, n_rounds+1))  for _ in range(n_players)]).flatten()
    list_players = np.array([[player] * n_rounds for player in players]).flatten()
    df_all = pd.DataFrame([[p, r] for p, r in zip(list_players, list_rounds)], columns=[COL_ID, COL_ROUND])
    
    df_merge = df_all.merge(df, how='left', on=[COL_ID, COL_ROUND])
    df_merge.fillna(value=0, inplace=True)
    
    x, y = [], []
    for r in range(from_round, to_round - last_rounds + 1):
        print('x: rodadas de {:}-{:} y: rodada {}'.format(r, r + last_rounds - 1, r + last_rounds))
        
        df_round = df_merge[(df_merge[COL_ROUND] >= r) & (df_merge[COL_ROUND] < r + last_rounds)]
        df_group = df_round.groupby(by=[COL_ID], as_index=False, sort=False)
        x.append([data[cols_of_interest].values for _, data in df_group])
            
        df_ids = pd.DataFrame([data[COL_ID].values[0] for _, data in df_group], columns=[COL_ID])
        if return_pred:
            df_next = df_merge[df_merge[COL_ROUND] == r + last_rounds]
            assert(np.all(df_ids[COL_ID].values ==  df_next[COL_ID].values))

            df_points = df_ids.merge(df_next, how='left', on=[COL_ID])[COL_POINTS]
            y.append(df_points.fillna(value=0).values)
        
    return np.concatenate([r for r in x]), np.array(y).flatten() if return_pred else df_ids[COL_ID].values
```

--------------------------------

### Importar Bibliotecas Essenciais para Análise de Dados do Cartola FC

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/colabs/caRtola_media_media_movel_media_casa_ou_fora_o_que_usar.ipynb

Importa bibliotecas Python necessárias para web scraping, manipulação de dados e visualização. Inclui re, requests, BeautifulSoup, numpy, pandas, seaborn e matplotlib.

```python
# Importar bibliotecas
import re                          # Expressão regulares
import requests                    # Acessar páginas da internet
from bs4 import BeautifulSoup      # Raspar elementos de páginas da internet
import numpy as np
import pandas as pd                # Abrir e concatenar bancos de dado
import seaborn as sns; sns.set()
import matplotlib.pyplot as plt
```

--------------------------------

### Create Dictionary of File URLs

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/colabs/caRtola_como_ler_repositório_do_github_com_BeautifulSoup_e_Pandas.ipynb

Iterates through HTML tags to find links matching the defined regex. It extracts the file name and constructs the raw file URL for each matching file, storing them in a dictionary.

```python
dict_of_files = {}
for tag in soup.find_all('a', attrs={'href': re.compile(regex)}):
    href_str = tag.get('href')
    file_name = re.sub('/henriquepgomide/caRtola/blob/master/data/2019/',
                       '',
                       href_str)
    
    file_url = re.sub('/henriquepgomide/caRtola/blob/master/data/2019/',
                    'https://raw.githubusercontent.com/henriquepgomide/caRtola/master/data/2019/',
                    href_str)

    dict_of_files[file_name] = file_url
```

--------------------------------

### Load 2019 Match Data (Python)

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/colabs/caRtola_media_media_movel_media_casa_ou_fora_o_que_usar.ipynb

Loads the match data for the 2019 season from a CSV file. This data is used to determine home and away teams for each match. The `head()` method displays the first few rows of the loaded data.

```python
# Abrir lista de partidas
partidas_2019 = pd.read_csv('https://raw.githubusercontent.com/henriquepgomide/caRtola/master/data/2019/2019_partidas.csv')
partidas_2019.head()
```

--------------------------------

### Fetch GitHub Repository Page

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/colabs/caRtola_como_ler_repositório_do_github_com_BeautifulSoup_e_Pandas.ipynb

Uses the requests library to download the HTML content of a specific GitHub repository page. This is the first step before parsing the HTML.

```python
# URL com caminho do repositório
URL = 'https://github.com/henriquepgomide/caRtola/tree/master/data/2019'
html = requests.get(URL)
```

--------------------------------

### Display first 5 rows of match data with normalized scores

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/markov-chain-lpp.ipynb

Shows the first five rows of the 'partidas' DataFrame, including the newly created normalized score columns.

```python
partidas.head()
```

--------------------------------

### Import Libraries and Configure Pandas

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/Análise dos Dados.ipynb

Imports essential libraries for data manipulation, numerical operations, plotting, and machine learning. Configures Pandas to display a maximum of 100 columns.

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import _pickle as pkl
from sklearn.model_selection import GridSearchCV
from sklearn.neural_network import MLPRegressor
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, MinMaxScaler

%matplotlib inline

pd.set_option('display.max_columns', 100)
```

--------------------------------

### Load and Inspect Data with Pandas

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/Análise dos Dados.ipynb

Loads a CSV file into a pandas DataFrame and prints its shape and the first 10 rows. Ensure the file path is correct.

```python
df = pd.read_csv('../../data/dados_agregados.csv')
print(df.shape)
df.head(10)
```

--------------------------------

### LightGBM Cross-Validation Results Display

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/desafio_valorizacao/Descobrindo o algoritmo de valorização do Cartola FC - Parte II.ipynb

Prints the current parameters used for LightGBM cross-validation, the best number of boosting rounds determined by early stopping, and the corresponding best cross-validation score.

```python
# Display results
print('Current parameters:\n', params)
print('\nBest num_boost_round:', len(cv_results['l2-mean']))
print('Best CV score:', cv_results['l2-mean'][-1])
```

--------------------------------

### Preparing Test Samples for Prediction

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/LSTM.ipynb

Creates input samples for the LSTM model from a DataFrame. Specify the columns of interest, the range of rounds to include, and the number of previous rounds to consider for each sample. Returns the prepared input data and corresponding IDs.

```python
x_test, ids = create_samples(df_samples, cols_of_interest, from_round=2, to_round=5, last_rounds=3, return_pred=False)

print(x_test.shape, ids.shape)
```

--------------------------------

### Saving the Trained Model

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/Análise dos Dados.ipynb

Serializes and saves the trained regression model 'reg' to a file using pickle. The model is saved to '../../src/python/models/nn1.pkl'.

```python
pkl.dump(reg, open('../../src/python/models/nn1.pkl', 'wb'), -1)
```

--------------------------------

### Generate Training Data by Round

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/Análise dos Dados.ipynb

Iterates through rounds to create training samples, appending them to a training DataFrame. Prints the number of players participating in each train/prediction round combination and the final training data dimensions.

```python
df_train = pd.DataFrame(data = [], columns=list(df_samples.columns) + ['pred'])
n_rounds = df_samples['Rodada'].max()

for round_train, round_pred in zip(range(1, n_rounds), range(2, n_rounds+1)):
    df_round = create_samples(df_samples, round_train, round_pred)
    print('qtde. de jogadores que participaram na rodada {0:=2} (train) e na rodada {1:=2} (pred): {2:=4}'.format(round_train, round_pred, df_round.shape[0]))
    df_train = df_train.append(df_round, ignore_index=True)
    
print("Dimensões dos dados de treinamento: ", df_train.shape)
```

--------------------------------

### Kedro Pipeline: Fill Empty Slugs

Source: https://context7.com/henriquepgomide/cartola/llms.txt

Generates a URL-friendly 'slug' for players based on their nickname if the 'slug' field is missing. It converts nicknames to lowercase, replaces spaces with hyphens, and removes accents using 'unidecode'.

```python
def fill_empty_slugs(df: pd.DataFrame) -> pd.DataFrame:
    """Gera slug a partir do apelido quando ausente (ex: 'João Silva' -> 'joao-silva')."""
    if "slug" not in df.columns:
        df["slug"] = None
    mask = df["slug"].isna()
    df.loc[mask, "slug"] = df.loc[mask, "apelido"].apply(
        lambda nick: unidecode(nick.lower().replace(" ", "-"))
    )
    return df
```

--------------------------------

### Create Kedro Pipeline for Merging

Source: https://context7.com/henriquepgomide/cartola/llms.txt

Defines a Kedro pipeline for the merge_datasets node, specifying inputs, outputs, namespace, and tags. The output 'concat' is mapped to 'preprocessing.raw'.

```python
def create_pipeline() -> Pipeline:
    return pipeline(
        [
            node(merge_datasets, inputs=["scouts", "players", "teams"], outputs="concat"),
        ],
        namespace="merge",
        tags=["merge"],
        outputs=dict(concat="preprocessing.raw"),  # Alimenta o pipeline de pré-processamento
    )
```

--------------------------------

### Join Player Ranks to Match Data

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/markov-chain-lpp.ipynb

Integrates player ranking information into the match dataframe by joining on player ID. Ensure 'atletas' and 'df_partidas' dataframes are pre-loaded and indexed appropriately.

```python
df_partidas = df_partidas.set_index('atletas.atleta_id').join(atletas.set_index('player_id'))
```

--------------------------------

### Display DataFrame Head

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/colabs/caRtola_como_ler_repositório_do_github_com_BeautifulSoup_e_Pandas.ipynb

Displays the first 5 rows and columns of a Pandas DataFrame. Useful for a quick overview of the data structure.

```python
cartola.head()
```

--------------------------------

### Load player statistics CSV

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/markov-chain-lpp.ipynb

Loads player statistics from a CSV file hosted on GitHub into a pandas DataFrame. Ensure the URL is correct and accessible.

```python
url = 'https://raw.githubusercontent.com/henriquepgomide/caRtola/master/data/2019/2019-medias-jogadores.csv'
medias = pd.read_csv(url)
```

--------------------------------

### Populate Transition Matrix

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/markov-chain-lpp.ipynb

Iterates through each match to update the transition matrix 'M'. It calculates scores based on player performance and game outcomes, then updates the matrix elements according to a defined rule involving home and away team scores and player scores.

```python
for partida in range(len(partidas)-1): #Vamos deixar a última partida de fora para testes
    df_rodada = df_partidas_posicao[df_partidas_posicao['round'] == partidas['round'][partida]]
    jogadores_casa = df_rodada[df_rodada['atletas.clube_id'] == partidas['home_team'][partida]]
    jogadores_visitantes = df_rodada[df_rodada['atletas.clube_id'] == partidas['away_team'][partida]]
    
    for j_casa in range(len(jogadores_casa)):
        for j_visitante in range(len(jogadores_visitantes)):
            score_casa = 0
            score_visitante = 0
            
            pontos_j_casa = jogadores_casa['atletas.pontos_num'].iloc[j_casa]
            pontos_j_visitante = jogadores_visitantes['atletas.pontos_num'].iloc[j_visitante]
            
            soma =  pontos_j_casa + pontos_j_visitante 
            if soma != 0:
                score_casa = pontos_j_casa / soma
                score_visitante = pontos_j_visitante / soma
            
            j1 = jogadores_casa['Rank'].iloc[j_casa]
            j2 = jogadores_visitantes['Rank'].iloc[j_visitante]
                
            M[j1,j1] = M[j1,j1] + partidas['home_score_norm'][partida] + score_casa
            M[j1,j2] = M[j1,j2] + partidas['away_score_norm'][partida] + score_visitante
            M[j2,j1] = M[j2,j1] + partidas['home_score_norm'][partida] + score_casa
            M[j2,j2] = M[j2,j2] + partidas['away_score_norm'][partida] + score_visitante
```

--------------------------------

### Train Linear Regression Model

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/desafio_valorizacao/Desafio da Valorização.ipynb

Initializes and trains a Linear Regression model using the preprocessed features (x) and target variable (y).

```python
reg = LinearRegression()
reg.fit(x, y)
```

--------------------------------

### Calcular Probabilidade de Time Não Levar Gols em R

Source: https://github.com/henriquepgomide/cartola/blob/master/src/R/tutoriais/tutorial-1-como-montamos-defesas-no-cartolafc-estatistica-modelagem.md

Este script em R calcula a probabilidade de cada time não levar gols com base em previsões de placares. Ele itera sobre as previsões, calcula a proporção de jogos onde o time não sofreu gols, e organiza os resultados em um data frame ordenado.

```r
home.team.names   <- names(teamPredictions$home.score)
away.team.names   <- names(teamPredictions$away.score)
home.goals.vector <- rep(NA,10)
away.goals.vector <- rep(NA,10)

for (i in 1:10){
  home.goals.vector[i] <- round(prop.table(table(teamPredictions$home.goals[i,] > 0))[2],2)
  away.goals.vector[i] <- round(prop.table(table(teamPredictions$away.goals[i,] > 0))[2],2)
}

team.names   <- c(home.team.names, away.team.names)
scoring.odds <- c(home.goals.vector, away.goals.vector)

scoring.odds.df <- data.frame(team.names = team.names, scoring.odds = scoring.odds*100)
scoring.odds.df <- arrange(scoring.odds.df, scoring.odds)

print(scoring.odds.df)
```

--------------------------------

### Display DataFrame Head

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/LSTM.ipynb

Displays the first 10 rows of the cleaned DataFrame.

```python
df_clean.head(10)
```

--------------------------------

### GridSearchCV Fitting Progress Output

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/Análise dos Dados.ipynb

Displays the progress and details of the GridSearchCV fitting process, including the number of fits and elapsed time.

```text
Fitting 5 folds for each of 3 candidates, totalling 15 fits
[CV] NN__hidden_layer_sizes=(50, 50, 50, 50) .........................
[CV] NN__hidden_layer_sizes=(50, 50, 50, 50) .........................
[CV] NN__hidden_layer_sizes=(50, 50, 50, 50) .........................
[CV] NN__hidden_layer_sizes=(50, 50, 50, 50) .........................
[CV]  NN__hidden_layer_sizes=(50, 50, 50, 50), score=-17.554050, total=   0.9s
[CV] NN__hidden_layer_sizes=(50, 50, 50, 50) .........................
[CV]  NN__hidden_layer_sizes=(50, 50, 50, 50), score=-20.194491, total=   1.1s
[CV] NN__hidden_layer_sizes=(50, 100, 50) .................לט
[CV]  NN__hidden_layer_sizes=(50, 50, 50, 50), score=-20.354199, total=   1.1s
[CV] NN__hidden_layer_sizes=(50, 50, 50, 50) .........................
[CV]  NN__hidden_layer_sizes=(50, 50, 50, 50), score=-17.402051, total=   1.2s
[CV] NN__hidden_layer_sizes=(50, 100, 50) .................לט
[CV]  NN__hidden_layer_sizes=(50, 100, 50), score=-18.117973, total=   0.6s
[CV] NN__hidden_layer_sizes=(50, 100, 50) .................לט
[CV]  NN__hidden_layer_sizes=(50, 100, 50), score=-20.206686, total=   1.1s
[CV] NN__hidden_layer_sizes=(50, 100, 50) .................לט
[CV]  NN__hidden_layer_sizes=(50, 100, 50), score=-20.316502, total=   1.0s
[CV] NN__hidden_layer_sizes=(50, 100, 50) .................לט
[CV] NN__hidden_layer_sizes=(50, 100, 100, 50) .......................
[CV]  NN__hidden_layer_sizes=(50, 50, 50, 50), score=-19.321397, total=   1.4s
[CV] NN__hidden_layer_sizes=(50, 100, 100, 50) .......................
[CV]  NN__hidden_layer_sizes=(50, 100, 100, 50), score=-20.332349, total=   1.7s
[CV] NN__hidden_layer_sizes=(50, 100, 100, 50) .......................
[CV]  NN__hidden_layer_sizes=(50, 100, 100, 50), score=-20.222579, total=   1.9s
[CV] NN__hidden_layer_sizes=(50, 100, 100, 50) .......................
[CV]  NN__hidden_layer_sizes=(50, 100, 50), score=-17.420738, total=   2.2s
[CV] NN__hidden_layer_sizes=(50, 100, 100, 50) .......................
[CV]  NN__hidden_layer_sizes=(50, 100, 50), score=-19.144452, total=   2.2s
[CV]  NN__hidden_layer_sizes=(50, 100, 100, 50), score=-19.244774, total=   0.8s
[CV]  NN__hidden_layer_sizes=(50, 100, 100, 50), score=-17.384773, total=   1.0s
[CV]  NN__hidden_layer_sizes=(50, 100, 100, 50), score=-18.157146, total=   1.3s
```

--------------------------------

### Download Cartola FC Market Data to CSV using Python

Source: https://context7.com/henriquepgomide/cartola/llms.txt

This script collects market data from the official Cartola FC API, processes it into a pandas DataFrame, standardizes column names, merges club information, and saves the data as a CSV file for a specific round. It avoids overwriting existing files.

```python
import os
from datetime import date
import pandas as pd
import requests

# Busca dados da API oficial do Cartola FC
data_json = requests.get("https://api.cartolafc.globo.com/atletas/mercado").json()

# Monta DataFrame de atletas expandindo a coluna "scout"
df_atletas = pd.DataFrame(data_json["atletas"])
df_atletas = df_atletas.join(pd.DataFrame(df_atletas.pop("scout").values.tolist()))

# Padroniza nome das colunas: colunas em minúsculo recebem prefixo "atletas."
df_atletas = df_atletas.rename(
    columns={col: f"atletas.{col}" if col.islower() else col for col in df_atletas.columns}
)

# Monta DataFrame de clubes e faz merge com atletas
df_clubes = pd.DataFrame(data_json["clubes"].values())
df_clubes = df_clubes.rename(columns=dict(id="atletas.clube_id", nome="atletas.clube.id.full.name"))
df_clubes = df_clubes[["atletas.clube_id", "atletas.clube.id.full.name"]]

df_merge = df_atletas.merge(df_clubes, how="left", on="atletas.clube_id")

# Ordena colunas: atributos do atleta primeiro, depois scouts (maiúsculas)
rodada = df_merge.loc[0, "atletas.rodada_id"].astype(str)
year = date.today().year
file = f"data/01_raw/{year}/rodada-{rodada}.csv"

cols_scouts = sorted([col for col in df_merge.columns if col.isupper()])
cols_atleta = sorted(list(set(df_merge.columns) - set(cols_scouts)))
df_merge = df_merge.loc[:, cols_atleta + cols_scouts]
df_merge.sort_values(by="atletas.atleta_id", inplace=True, ignore_index=True)

# Salva somente se o arquivo ainda não existir
if not os.path.exists(file):
    os.makedirs(os.path.dirname(file), exist_ok=True)
    df_merge.to_csv(file, index=False)
    print(f"Arquivo salvo: {file}")
else:
    print(f"Arquivo já existe: {file}")
```

--------------------------------

### Display column names of player data

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/markov-chain-lpp.ipynb

Lists all the column names present in the 'medias' DataFrame. Helps in identifying available features.

```python
medias.columns
```

--------------------------------

### GridSearchCV for Neural Network Hyperparameter Tuning

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/Análise dos Dados.ipynb

Tunes hyperparameters for a neural network model using GridSearchCV. Requires a pipeline 'pipe', parameter grid 'params', and training data 'samples' and 'scores'.

```python
params = dict(NN__hidden_layer_sizes=[(50,50,50,50), (50,100,50), (50,100,100,50)])

reg = GridSearchCV(pipe, params, scoring='neg_mean_squared_error', n_jobs=-1, cv=5, verbose=10)
reg.fit(samples, scores)
print(reg.best_params_, reg.best_score_)
```

--------------------------------

### Load R Packages and Match Data

Source: https://github.com/henriquepgomide/cartola/blob/master/src/R/tutoriais/tutorial-1-como-montamos-defesas-no-cartolafc-estatistica-modelagem.md

Loads necessary R packages for data manipulation and statistical modeling. Reads CSV data for football matches from a specified path.

```r
# Carregar pacotes
library(tidyverse) # Manipulação dos dados
library(fbRanks)   # Uso do modelo de Dixon e Coles para estimativa da força de um time

# Abrir dados dos confrontos
matches <- read.csv("~/caRtola/data/2018/2018_partidas.csv", stringsAsFactors = FALSE)
```

--------------------------------

### Display First Rows of Merged Data

Source: https://github.com/henriquepgomide/cartola/blob/master/notebooks/markov-chain-lpp.ipynb

Shows the first few rows of the 'df_partidas' dataframe after merging with player ranking data. Useful for verifying the join operation and inspecting the new columns.

```python
df_partidas.head()
```