#!/bin/ksh ## ## SCRIPT NAME: select_subdir_siz_recs_division_bygui ## ## Where: in $FEDIR/scripts where $FEDIR=/apps/nns_com/fea ## ############################################################################## ## PURPOSE: This script ## creates a SIZE-SORTED (or NAME-SORTED) SUB-DIRECTORY-SIZES LISTING ## --- ## by EXTRACTING data-lines from a very large file (~10,000 lines), ## created weekly. ## ## The input & output data-lines contain a directory-name and its size. ## ##--- ## THE EXTRACT: ## ## A GUI is presented that allows one to EXTRACT data-lines with ## 1) a max of N-directory-levels in the directory-name ## AND/OR ## 2) by string-match in the directory name. ## ##--- ## THE INPUT FILE: ## ## The large weekly-created file has a filename like ## ## ${REPORTDIR}/${REPORTFILE_BASE} ## ## where ## REPORTDIR="/data/cvn78/divbatch/catia_files/logs" ## in file-system engprd00:/data ## ## and REPORTFILE_BASE="spac_div_presentations_dirs_siz.lis" ## and = 'prev' or 'prev2' or 'presort'. ## ############################################################################## ## EXTRACT EXAMPLES: ## ## The extract is done in either of two ways ## ## 1) specifying a max-number of DIRECTORY LEVELS to select&show ## (1,2,3,4 or 0=all) ## ## 2) specifying a DIR-STRING to search for. Examples: ## ## /division/NNSEB (a sub-directory name) ## /TWB_ or ER or /models (part of a sub-directory name) ## bem or TWB (part of a userid, in subdir name) ## ## OR 3) a combination of the two, DIR-LEVS & DIR-STRING. ## ############################################################################## ## SOURCE OF THE ALL-SUBDIRECTORIES REPORT FILE(S): ## ## The input file ## ## 'spac_div_presentations_dirs_siz.lis' ## ## is generated --- on the SGI File Server engvis00 --- ## by a script (run in a weekly cron job). ## ## The cron script is ## ## 'divreport_subdir_sizes_engvis00.scr' ## ## in directory '/apps/autobatch/clean_scripts_divadm'. ## ## See that cron script for details. ## ## The cron script is meant to be called by a 'divadm' ## crontab job on 'engvis00' --- say, weekly. ## ## See crontab file 'CRONTAB_DIV_USERDATA_UTILS_engvis00' ## in directory '/apps/autobatch'. ## ## NOTE: That cron-script makes 'header' & 'trailer' lines of ## the cron-report with a '#' character in column 1. So ... ## ## The data can be separated from the 'header' & 'trailer' lines ## (and the null lines) using ## ## grep -v "^#" $DIRRPT | grep -v "^$" ## ############################################################################## ## CALLED BY: toolchest script 'divbat_tools' in $FEDIR/scripts ## actually, by 'divbat_tools.chestdef' in $FEDIR/scripts. ## ## The toolchest-drawer hierarchy is something like ## ## SGI Toolchest -> ## Division -> ## 'divbat' toolchest -> ## User Presentation Directory tools -> ## /division SUBDIR-SIZES EXTRACT Utility (from weekly snapshot) ## ############################################################################## ## MAINTENANCE HISTORY: ## Written by: Blaise Montandon 22aug2003 Based on script ## 'select_subdir_siz_recs_bygui' ## in $FEDIR/scripts. ## Updated by: Blaise Montandon 22aug2003 Add the SORT_TYPE variable ## and an 'xconfirm' prompt for its ## value. ############################################################################## ################################################################### ## Set some hostnames --- for extract-report header & trailer. ################################################################### THISHOST=`hostname` # MAIN_SGI_SERVER="engprd00" VIZ_SGI_SERVER="engvis00" QUERY_HOST="$VIZ_SGI_SERVER" ################################################################### ## SET VARS THAT SPECIFY THE DIRECTORY (FILE-SYSTEM) OF INTEREST ## --- and the name of the 'du' report file. ################################################################### QUERY_DIR="/division" REPORTDIR="/data/cvn78/divbatch/catia_files/logs" REPORTFILE_BASE="spac_div_presentations_dirs_siz.lis" ############################################################################ ## Set scripts pathname in case this toolchest is not started from ## the nnsFEAmenu system --- and a $FEDIR utility script (or help) is needed. ############################################################################ if test "$FEDIR" = "" then FEDIR="/apps/nns_com/fea" fi ######################################################################## ## SET TITLE FOR GUI WINDOW(S). ######################################################################## WINTITLE="SubDir Size List, Extract Info" export WINTITLE ######################################################################## ## COULD ALSO INITIALIZE GUI QUERY PARMS, ## different from the defaults built into the GUI. ######################################################################## # DIRRPT="${REPORTDIR}/${REPORTFILE_BASE}_prev" DIRRPT="${REPORTDIR}/$REPORTFILE_BASE" # DIRLEV=5 DIRLEV=3 SEARCHSTR="${QUERY_DIR}/NNSEB" # SEARCHSTR="" export DIRRPT DIRLEV SEARCHSTR SUBDIR_UTIL_USAGE_INFO="\ The Directory search string is optional. Clear it quickly with the 'Clear' button. If you set DirLevs=0 (Search ALL dir levels), a search string should be specified -- otherwise, you get the entire directory report --- which is huge ... many 1,000's of directories. This extract could then take about a minute. You can paste an example 'directory Search-String', from below, to the Search-String entry field above. _______________________________________________________________________________ EXAMPLES FOR '$QUERY_DIR' QUERIES from report files ${REPORTDIR}/${REPORTFILE_BASE} or ${REPORTDIR}/${REPORTFILE_BASE}_prev or ${REPORTDIR}/${REPORTFILE_BASE}_prev2 DirLevs SearchString Shows sizes of sub-directories like ------- ----------------- --------------------------------------------- 2 NONE ${QUERY_DIR} and ${QUERY_DIR}/ 3 NONE ${QUERY_DIR} and ${QUERY_DIR}/ and ${QUERY_DIR}// 3 /NNS ${QUERY_DIR}/NNS* and ${QUERY_DIR}/NNS*/ and ${QUERY_DIR}//NNS* 3 /NNSEB/ ${QUERY_DIR}/NNSEB/ or ${QUERY_DIR}/NNSEB/ 4 /NNSEB/ ${QUERY_DIR}/NNSEB/ and ${QUERY_DIR}/NNSEB// and ${QUERY_DIR}//NNSEB/ 4 ${QUERY_DIR}/NNSEB/ ${QUERY_DIR}/NNSEB/ and ${QUERY_DIR}/NNSEB// " export SUBDIR_UTIL_USAGE_INFO ######################################################################## ## PROMPT FOR QUERY PARMS -- cron-SUBDIRS-SIZE-LISTING, DIR-LEVELS, ## and (optional) DIR-SEARCH-STRING. ######################################################################## ## NOTE: Could activate the while-loop when the GUI, enter_subdir_queries.tk, ## is updated to use the DIRRPT, DIRLEVS, SEARCHSTR env vars ## to re-initialize the GUI to current settings. ######################################################################## ## while true while : do TEMP=`$FEDIR/tkGUIs/enter_subdir_queries.tk` eval "$TEMP" ## TEMP contains a string like ## DIRRPT=""; DIRLEVS=""; SEARCHSTR="" ## For testing: # echo "$TEMP" if test "$DIRRPT" = "" then exit fi ##################################################################### ## CHECK THAT THE REPORT FILE ENTERED EXISTS. ##################################################################### if test ! -f $DIRRPT then CONFIRM_TEXT="\ Specified Directory-Report Filename: $DIRRPT not found or does not exist. Some candidates are ${REPORTDIR}/${REPORTFILE_BASE} ${REPORTDIR}/${REPORTFILE_BASE}_prev ${REPORTDIR}/${REPORTFILE_BASE}_prev2 A search of these files will take on the order of 20 secs to a minute. Exiting. " export CONFIRM_TEXT CONFIRM_GEOM="+080+080" CONFIRM_MINSIZE="350 150" CONFIRM_TITLE="$WINTITLE" export CONFIRM_TITLE CONFIRM_MINSIZE CONFIRM_GEOM CONFIRM_COLOR="#3DADFF" export CONFIRM_COLOR CONFIRM_FONT="-adobe-helvetica-bold-r-normal--*-100-*-*-p-*-*-*" export CONFIRM_FONT JUNK=`$FEDIR/tkGUIs/confirm.tk DISMISS` & ## exit export DIRRPT DIRLEVS SEARCHSTR continue fi ## END OF if test ! -f $DIRRPT ##################################################################### ## At this point, 'eval $TEMP' should have set the three vars --- ## DIRRPT DIRLEVS SEARCHSTR. Example: ## ## DIRRPT="/data/cvn78/divbatch/catia_files/logs/spac_div_presentations_dirs_siz.lis"; ## DIRLEVS="3"; ## SEARCHSTR="" ##################################################################### ## EXIT if DIRLEVS & SEARCHSTR are null (user cancelled query) ##################################################################### if test \( "$DIRLEVS" = "" -a "$SEARCHSTR" = "" \) then exit fi ############################################################################## ## PROMPT USER WHETHER SORT should be by dirname or dirsize. ## (Not implemented at this time.) ############################################################################## SORT_TYPE="SIZE" SORT_TYPE=`xconfirm -c -header "SubDir Size Report Utility" \ -B SIZE -b NAME \ -t "Do you want to see the lines of the SubDir-Sizes Report sorted by" \ -t "subdir-SIZE or by subdir-NAME?" \ -t "" \ -t " SIZE or NAME" \ -icon warning` # # SORTCMD="" # # if test "$SORT_TYPE" = "NAME" # # then # # SORTCMD=" | sort -k2" # # fi ##################################################################### ## PREP TEXT OUTPUT LIST FILE -- and a temp file. ##################################################################### . $FEDIR/scripts/set_localoutlist OUTLIST2=${OUTLIST}_temp ##################################################################### ## PREP HEADER FOR LIST FILE. Cases considered: ## 1) SEARCHSTR not-blank & DIRLEVS = 0 (ALL) ## 2) SEARCHSTR not-blank & DIRLEVS > 0 ## 3) SEARCHSTR blank & DIRLEVS > 0 (up to DIRLEVS subdir levs of DIRRPT) ## 4) SEARCHSTR blank & DIRLEVS = 0 (all subdirs of the DIRRPT) ## Also add 1st rec of report file (date) to the Header. ##################################################################### if test \( ! "$SEARCHSTR" = "" -a "$DIRLEVS" = "0" \) then SEARCH_CRIT="\ ****************************************************** Max-Dir-Levs: $DIRLEVS Search-string: $SEARCHSTR ****************************************************** (I.e. into this report, EXTRACT ALL sub-directory size records that contain the search string.)" fi if test \( ! "$SEARCHSTR" = "" -a ! "$DIRLEVS" = "0" \) then SEARCH_CRIT="\ ****************************************************** Max-Dir-Levs: $DIRLEVS Search-string: $SEARCHSTR ****************************************************** (I.e. into this report, EXTRACT sub-directory size records that contain the search string and no more than $DIRLEVS directory levels.)" fi if test \( "$SEARCHSTR" = "" -a "$DIRLEVS" = "0" \) then SEARCH_CRIT="\ ****************************************************** Max-Dir-Levs: $DIRLEVS Search-string: $SEARCHSTR ****************************************************** I.e. Show ALL sub-directory size recs in this report. No extract. This is essentially the same as simply looking at the original 'cron' report --- but with more readable formatting of the numbers (subdir sizes). PROBABLY HUGE. DO NOT PRINT." fi if test \( "$SEARCHSTR" = "" -a ! "$DIRLEVS" = "0" \) then SEARCH_CRIT="\ ****************************************************** Max-Dir-Levs: $DIRLEVS Search-string: $SEARCHSTR ****************************************************** (I.e. into this report, EXTRACT sub-directory size records with no more than $DIRLEVS directory levels.)" fi DATELINE=`head -1 $DIRRPT` echo "\ ................... `date '+%Y %b %d %a %T%p %Z'` ...................... ******************************** DISK USAGE of SUB-DIRECTORIES of '$QUERY_DIR'. ******************************** ( SORTED BY ** $SORT_TYPE ** ) AN EXTRACT FROM 'cron' REPORT file $DIRRPT which was generated mid-week. **** First line of 'cron' report file --- *DATE* that cron job was run: **** $DATELINE ================================== PARAMETERS FOR THIS report-EXTRACT: ================================== $SEARCH_CRIT Disk usage Subdirectory (Megabytes) name ------------- ------------ TerGigMeg.Kil | | | | " > $OUTLIST ###################################################################### ## Run 'awk' to select recs for report -- using DIRLEVS and SEARCHSTR. ###################################################################### ## IF DIRLEVS NOT = 0, Set a DIRMASK according to # of DIRLEVS ## requested -- like DIRMASK=/.*/.*/ (Three slashes) when DIRLEVS=2 ## -- and put the non-comment, non-blank recs of the $DIRRPT file ## that match that DIRMASK ## into the temp OUTLIST file, $OUTLIST2. ##################################################################### ## ELSE (IF DIRLEVS = 0), put ALL the non-comment, non-blank recs ## of the $DIRRPT file into the temp OUTLIST file, $OUTLIST2. ##################################################################### ## In either case, right-justify the subdir size-numbers in the 1st col, ## using awk and format the sizes in Megabytes instead of Kbytes. ##################################################################### if test "$SEARCHSTR" = "NONE" then SEARCHSTR="" fi if test ! "$DIRLEVS" = "0" then ###################################################################### ## Build $DIRMASK to use in 'grep -v' that removes dir-report-lines ## that have *MORE THAN* the number of slashes indicated by $DIRLEVS. ###################################################################### ## For example: ' grep -v "/.*/.*/" ' will throw away all dirnames ## with 3 or more slashes --- leaving names like /xxx and /xxx/yyy. ###################################################################### DIRMASK="/" COUNT=0 while test $COUNT -lt $DIRLEVS do DIRMASK=${DIRMASK}".*/" COUNT=`expr $COUNT + 1` done if test "$SEARCHSTR" = "" then if test "$SORT_TYPE" = "SIZE" then grep -v "^#" $DIRRPT | grep -v "^$" | grep -v "$DIRMASK" | \ awk \ '{printf ("%13.3f %s \n", $1/1000, $2 )}' \ >> $OUTLIST else grep -v "^#" $DIRRPT | grep -v "^$" | grep -v "$DIRMASK" | \ sort -k2 | awk \ '{printf ("%13.3f %s \n", $1/1000, $2 )}' \ >> $OUTLIST fi else if test "$SORT_TYPE" = "SIZE" then grep -v "^#" $DIRRPT | grep -v "^$" | grep -v "$DIRMASK" | \ awk -v SEL_STR="$SEARCHSTR" \ '{ if ( $0 ~ SEL_STR ) {printf ("%13.3f %s \n", $1/1000, $2 )}}' \ >> $OUTLIST else grep -v "^#" $DIRRPT | grep -v "^$" | grep -v "$DIRMASK" | \ sort -k2 | awk -v SEL_STR="$SEARCHSTR" \ '{ if ( $0 ~ SEL_STR ) {printf ("%13.3f %s \n", $1/1000, $2 )}}' \ >> $OUTLIST fi fi ## END OF if test "$SEARCHSTR" = "" ###################################################################### ## *OLD* - BEFORE CONVERSION OF KBYTES TO GIG AND BEFORE USE OF SEL_STR. # awk '{printf ("%11s %s \n", $1, $2 )}' > $OUTLIST2 ###################################################################### else ###################################################################### ## Consider *ALL* the dir-report-lines for the output. ## I.e. no 'grep -v' using $DIRMASK. ###################################################################### if test "$SEARCHSTR" = "" then if test "$SORT_TYPE" = "SIZE" then grep -v "^#" $DIRRPT | grep -v "^$" | \ awk \ '{printf ("%13.3f %s \n", $1/1000, $2 )}' \ >> $OUTLIST else grep -v "^#" $DIRRPT | grep -v "^$" | \ sort -k2 | awk \ '{printf ("%13.3f %s \n", $1/1000, $2 )}' \ >> $OUTLIST fi else if test "$SORT_TYPE" = "SIZE" then grep -v "^#" $DIRRPT | grep -v "^$" | \ awk -v SEL_STR="$SEARCHSTR" \ '{ if ( $0 ~ SEL_STR ) {printf ("%13.3f %s \n", $1/1000, $2 )}}' \ >> $OUTLIST else grep -v "^#" $DIRRPT | grep -v "^$" | \ sort -k2 | awk -v SEL_STR="$SEARCHSTR" \ '{ if ( $0 ~ SEL_STR ) {printf ("%13.3f %s \n", $1/1000, $2 )}}' \ >> $OUTLIST fi fi ## END OF else IN if test "$SEARCHSTR" = "" ###################################################################### ## OLD - BEFORE CONVERSION OF KBYTES TO GIG AND BEFORE USE OF SEL_STR. # awk '{printf ("%11s %s \n", $1, $2 )}' > $OUTLIST2 ###################################################################### fi ## END OF if test ! "$DIRLEVS" = "0" ##################################################################### ## NOT NEEDED ANY MORE. PUT SELECT-FUNCTION IN THE 'awk' above. ##################################################################### # ##################################################################### # ## IF SEARCHSTR NOT = blank, extract search string recs # ## from OUTLIST2, which contains a specified number of DIRLEVS recs. # ##################################################################### # # if test ! "$SEARCHSTR" = "" # then # ## Remove any blanks in the search-string (esp. trailing blanks) # ## -- used before 'string-trim' of DIRRPT was put in the GUI. # # SEARCHSTR=`echo "$SEARCHSTR" | sed 's| ||g'` # # mv $OUTLIST2 $OUTLIST # grep "$SEARCHSTR" $OUTLIST > $OUTLIST2 # fi # # ################################################# # ## ADD THE REPORT SUBDIR SIZE RECS (in OUTLIST2) # ## TO THE HEADER (in OUTLIST). # ################################################# # # cat $OUTLIST2 >> $OUTLIST # # rm $OUTLIST2 ##################################################################### ################################################# ## ADD TRAILER TO THE REPORT. ################################################# echo " | | | | TerGigMeg.Kil ------------- ------------ Disk usage Subdirectory (Megabytes) name ................... `date '+%Y %b %d %a %T%p %Z'` ...................... ******************************** DISK USAGE of SUB-DIRECTORIES of '$QUERY_DIR'. ******************************** ( SORTED BY ** $SORT_TYPE ** ) ================================== PARAMETERS FOR THIS report-EXTRACT: ================================== ****************************************************** Max-Dir-Levs: $DIRLEVS Search-string: $SEARCHSTR ****************************************************** NOTE: If this extract-and-sort is not quite to your liking, you can EDIT this report file (or a copy of it) --- removing lines and re-grouping remaining lines and inserting blank lines. Then you can (browse and) PRINT the result with 'xpg'. ----------------------------- SOURCE OF THIS REPORT EXTRACT: ----------------------------- This *EXTRACT* was obtained from a list of 1,000's of subdirectory sizes in file $DIRRPT The 'cron' job that built that report file essentially did the command du -k $QUERY_DIR | sort +0 -1 -nr on the host, $QUERY_HOST, where the '$QUERY_DIR' directory 'resides'. To reduce load on the system and to get quick response, THIS REPORT IS EXTRACTED FROM one of the weekly CRON JOB REPORT LISTINGs done on host $QUERY_HOST , late at night. ----------------- OTHER INPUT FILES: ----------------- The complete sub-directory size listings of '$QUERY_DIR' [ 1000's of sub-directories ] can be seen by browsing any of the following 3 files using the site 'xpg' command ( or 'nedit -read' ) ${REPORTDIR}/${REPORTFILE_BASE} ${REPORTDIR}/${REPORTFILE_BASE}_prev ${REPORTDIR}/${REPORTFILE_BASE}_prev2 These are the latest 'du' output --- and one week previous, and two-weeks previous. The run date-times are indicated at the top of those files. ------------------------------------- HOW TO GENERATE THIS REPORT *EXTRACT*: ------------------------------------- This EXTRACT was built by the script $0 which can be run from a GUI 'toolchest' via the site toolchests-drawers: SGI Toolchest -> Division -> 'divbat' toolchest -> User Presentation Directory tools -> /division SUBDIR-SIZES EXTRACT Utility (from weekly snapshot) --------- ----------------------- MORE HELP for the EXTRACT UTILITY: --------------------------------- You can see more information on running this type of report-extract utility by using the 'Help' button at the bottom of the 'SubDir Size List, Extract Info' GUI. That Help option shows a help file like $FEDIR/helps/subdir_queries.hlp ........................................................................... " >> $OUTLIST ##################################################################### ## SHOW EXTRACT REPORT. ##################################################################### ## FOR TESTING: # set -x # export SHOFILENAME="$OUTLIST" # $FEDIR/scripts/shofil $OUTLIST $FEDIR/tkGUIs/shofil.tk $OUTLIST & ## FOR TESTING: # set - ##################################################################### ## SET ENTRY FIELDS (QUERY PARMS) FOR NEXT GUI PROMPT. ##################################################################### # DIRRPT= # DIRLEVS= # SEARCHSTR= export DIRRPT DIRLEVS SEARCHSTR done ##################################################################### ## END OF PROMPTING LOOP, for cron-SUBDIRS-SIZE-LISTING, DIR-LEVELS, ## and (optional) DIR-SEARCH-STRING. #####################################################################