feature_builder module
--
-
- -class feature_builder.FeatureBuilder(input_df: DataFrame, vector_directory: str = './vector_data/', output_file_base: str = 'output', output_file_path_chat_level: str = None, output_file_path_user_level: str = None, output_file_path_conv_level: str = None, custom_features: list = [], analyze_first_pct: list = [1.0], turns: bool = False, conversation_id_col: str = 'conversation_num', speaker_id_col: str = 'speaker_nickname', message_col: str = 'message', timestamp_col: str | tuple[str, str] = 'timestamp', timestamp_unit='ms', grouping_keys: list = [], cumulative_grouping=False, within_task=False, ner_training_df: DataFrame = None, ner_cutoff: int = 0.9, regenerate_vectors: bool = False, compute_vectors_from_preprocessed: bool = False, custom_liwc_dictionary_path: str = '', convo_aggregation=True, convo_methods: list = ['mean', 'max', 'min', 'stdev'], convo_columns: list = None, user_aggregation=True, user_methods: list = ['mean', 'max', 'min', 'stdev'], user_columns: list = None) -
Bases:
-objectThe FeatureBuilder is the main engine that reads in the user’s inputs and specifications and generates -conversational features. The FeatureBuilder separately calls the classes -(ChatLevelFeaturesCalculator, ConversationLevelFeaturesCalculator, and -UserLevelFeaturesCalculator) to generate conversational features at different levels.
--
-
- Parameters: -
-
-
input_df (pd.DataFrame) – A pandas DataFrame containing the conversation data that you wish to featurize.
-vector_directory (str) – Directory path where the vectors are to be cached. Defaults to “./vector_data/”.
-output_file_base (str) – Base name for the output files, used to auto-generate filenames for each -of the three levels. Defaults to “output.”
-output_file_path_chat_level (str) – Path where the chat (utterance)-level output csv file is -to be generated. This parameter will override the base name.
-output_file_path_user_level (str) – Path where the user (speaker)-level output csv file is -to be generated. This parameter will override the base name.
-output_file_path_conv_level (str) – Path where the conversation-level output csv file is to be -generated. This parameter will override the base name.
-custom_features (list, optional) – A list of additional features outside of the default features that should -be calculated. Defaults to an empty list (i.e., no additional features beyond the defaults will -be computed).
-analyze_first_pct (list(float), optional) – Analyze the first X% of the data. This parameter is useful because the -earlier stages of the conversation may be more predictive than the later stages. Defaults to [1.0].
-turns (bool, optional) – If true, collapses multiple “chats”/messages by the same speaker in a row into a -single “turn.” Defaults to False.
-conversation_id_col (str, optional) – A string representing the column name that should be selected as -the conversation ID. Defaults to “conversation_num”.
-speaker_id_col (str, optional) – A string representing the column name that should be selected as the speaker ID. -Defaults to “speaker_nickname”.
-message_col (str, optional) – A string representing the column name that should be selected as the message. -Defaults to “message”.
-timestamp_col (str, optional) – A string representing the column name that should be selected as the message. -Defaults to “timestamp”.
-timestamp_unit (str, optional) – A string representing the unit of the timestamp (if the timestamp is numeric). -Defaults to ‘ms’ (milliseconds). Other options (D, s, ms, us, ns) can be found on the Pandas -reference: https://pandas.pydata.org/docs/reference/api/pandas.to_datetime.html
-grouping_keys (list, optional) – A list of multiple identifiers that collectively identify a conversation. If -non-empty, the data will be grouped by all keys in the list and use the grouped key as the unique -“conversational identifier.”
-cumulative_grouping (bool, optional) – If true, uses a cumulative way of grouping chats (looking not just within -a single ID, but also at what happened before). NOTE: This parameter and the following one -(within_grouping) were created in the context of a multi-stage Empirica game (see: -https://github.com/Watts-Lab/multi-task-empirica). Assumes exactly 3 nested columns at different -levels: a High, Mid, and Low level; that are temporally nested. Defaults to False.
-within_task (bool, optional) – If true, groups cumulatively such that only prior chats of the same “task” -(Mid-level identifier) are considered. Defaults to False.
-ner_training_df (pd.DataFrame, optional) – A pandas DataFrame of training data for named entity recognition features. -Defaults to None and will not generate named entity features if it does not exist.
-ner_cutoff (int) – The cutoff value for the confidence of prediction for each named entity. -Defaults to 0.9.
-regenerate_vectors (bool, optional) – If true, regenerates vector data even if it already exists. Defaults to False.
-compute_vectors_from_preprocessed (bool, optional) – If true, computes vectors using preprocessed text (with -capitalization and punctuation removed). Defaults to False.
-custom_liwc_dictionary_path (str, optional) – This is the path of the user’s own LIWC dictionary file (.dic). Defaults to empty string.
-convo_aggregation (bool, optional) – If true, aggregates features at the conversational level. Defaults to True.
-convo_methods (list, optional) – Specifies which aggregation functions (e.g., mean, stdev) to use at the -conversational level. Defaults to [‘mean’, ‘max’, ‘min’, ‘stdev’].
-convo_columns (list, optional) – Specifies which columns (at the utterance/chat level) to aggregate for the -conversational level. Defaults to all numeric columns.
-user_aggregation (bool, optional) – If true, aggregates features at the speaker/user level. Defaults to True.
-user_methods (list, optional) – Specifies which functions to aggregate with (e.g., mean, stdev) at the user level. -Defaults to [‘mean’, ‘max’, ‘min’, ‘stdev’].
-user_columns (list, optional) – Specifies which columns (at the utterance/chat level) to aggregate for the -speaker/user level. Defaults to all numeric columns.
-
-- Returns: -
The FeatureBuilder writes the generated features to files in the specified paths. The progress -will be printed in the terminal, indicating completion with “All Done!”.
-
-- Return type: -
None
-
-
-
-
- -chat_level_features() None -
Instantiate and use the ChatLevelFeaturesCalculator to create chat-level features.
-This function creates chat-level features using the ChatLevelFeaturesCalculator -and adds them to the self.chat_data dataframe. It also removes special -characters from the column names.
--
-
- Returns: -
None
-
-- Return type: -
None
-
-
-
-
- -conv_level_features() None -
Instantiate and use the ConversationLevelFeaturesCalculator to create conversation-level features.
-This function creates conversation-level features using -the ConversationLevelFeaturesCalculator, and adds them to the self.conv_data dataframe.
--
-
- Returns: -
None
-
-- Return type: -
None
-
-
-
-
- -featurize() None -
Main driver function for feature generation.
-This function creates chat-level features, generates features for different -truncation percentages of the data if specified, and produces user-level and -conversation-level features. Finally, the features are saved into the -designated output files.
--
-
- Returns: -
None
-
-- Return type: -
None
-
-
-
-
- -get_first_pct_of_chat(percentage) None -
Truncate each conversation to the first X% of rows.
-This function groups the chat data by conversation_num and retains only -the first X% of rows for each conversation.
--
-
- Parameters: -
percentage (float) – Percentage of rows to retain in each conversation
-
-- Returns: -
None
-
-- Return type: -
None
-
-
-
-
- -merge_conv_data_with_original() None -
Merge conversation-level data with the original dataset, so that we retain all desired columns, both original and generated.
-If the conversational identifier was generated by our system (as opposed to an existing column in the original dataset), -the function uses the preprocessed data, which contains the generated ID. Otherwise, it uses the original dataset.
-The function groups the original conversation data by “conversation_num” and merges it with the -conversation-level data (conv_data). It drops duplicate rows and removes the ‘index’ column if present.
--
-
- Returns: -
None
-
-- Return type: -
None
-
-
-
-
- -preprocess_chat_data() None -
Call all preprocessing modules needed to clean the chat text.
-This function groups the chat data as specified, verifies column presence, creates original and lowercased columns, preprocesses text, and optionally processes chat turns.
--
-
- Parameters: -
-
-
turns (bool, optional) – Whether to preprocess naive turns, defaults to False
-col – Columns to preprocess, including conversation_id, speaker_id and message, defaults to None
-within_task (bool, optional) – Whether to group within tasks, defaults to False
-
-- Returns: -
None
-
-- Return type: -
None
-
-
-
-
- -save_features() None -
Save the feature dataframes to their respective output file paths.
-This function saves the chat_data, user_data, and conv_data dataframes -to the respective CSV files specified in the output file paths provided during initialization.
--
-
- Returns: -
None
-
-- Return type: -
None
-
-
-
-
- -set_self_conv_data() None -
Derives the base conversation level dataframe.
-Set Conversation Data around conversation_num once preprocessing completes. -We need to select the first TWO columns, as column 1 is the ‘index’ and column 2 is ‘conversation_num’.
--
-
- Returns: -
None
-
-- Return type: -
None
-
-
-
-
- -user_level_features() None -
Instantiate and use the UserLevelFeaturesCalculator to create user-level features.
-This function creates user-level features using -the UserLevelFeaturesCalculator, and adds them to the self.user_data dataframe. -It also removes special characters from the column names.
--
-
- Returns: -
None
-
-- Return type: -
None
-
-
feature_builder module
C
CC |
- + |
F
|
